Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Constrain attrs (mac) and opencv (linux) in 1.3.4 #1927

Closed
wants to merge 13 commits into from
Closed

Constrain attrs (mac) and opencv (linux) in 1.3.4 #1927

wants to merge 13 commits into from

Conversation

roomrys
Copy link
Collaborator

@roomrys roomrys commented Aug 30, 2024 •
edited
Loading

Description

This PR is a hotfix to add further constraints to attrs (<22.2.0) and opencv (<=4.8.1 # [linux]).

Items

Manual Installation Tests

Linux

GUI Training

image
image

Terminal: Training 
Resetting monitor window.
Polling: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105332.centroid.n=100/viz/validation.*.png
Start training centroid...
['sleap-train', '/tmp/tmpd1et2_6d/240903_105333_training_job.json', '/home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/courtship_labels.slp', '--zmq', '--save_viz']
INFO:sleap.nn.training:Versions:
SLEAP: 1.3.4
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Linux-5.15.0-78-generic-x86_64-with-debian-bookworm-sid
INFO:sleap.nn.training:Training labels file: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Training profile: /tmp/tmpd1et2_6d/240903_105333_training_job.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
    "training_job_path": "/tmp/tmpd1et2_6d/240903_105333_training_job.json",
    "labels_path": "/home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/courtship_labels.slp",
    "video_paths": [
        ""
    ],
    "val_labels": null,
    "test_labels": null,
    "base_checkpoint": null,
    "tensorboard": false,
    "save_viz": true,
    "zmq": true,
    "run_name": "",
    "prefix": "",
    "suffix": "",
    "cpu": false,
    "first_gpu": false,
    "last_gpu": false,
    "gpu": "auto"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
    "data": {
        "labels": {
            "training_labels": "/home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/courtship_labels.slp",
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": [
                99,
                13,
                36,
                84,
                5,
                29,
                16,
                11,
                53,
                88,
                9,
                66,
                82,
                79,
                78,
                77,
                91,
                50,
                35,
                37,
                75,
                26,
                69,
                31,
                62,
                8,
                97,
                32,
                67,
                59,
                85,
                2,
                83,
                87,
                1,
                12,
                42,
                21,
                65,
                68,
                48,
                90,
                95,
                25,
                49,
                4,
                63,
                89,
                98,
                73,
                86,
                60,
                94,
                24,
                47,
                93,
                80,
                0,
                17,
                51,
                30,
                23,
                38,
                71,
                18,
                39,
                44,
                81,
                10,
                20,
                57,
                6,
                15,
                64,
                61,
                22,
                56,
                54,
                46,
                27,
                72,
                14,
                55,
                41,
                70,
                45,
                74,
                76,
                92,
                34
            ],
            "validation_inds": [
                43,
                96,
                7,
                58,
                28,
                19,
                40,
                33,
                3,
                52
            ],
            "test_inds": null,
            "search_path_hints": [
                "",
                "",
                "",
                "",
                "",
                ""
            ],
            "skeletons": []
        },
        "preprocessing": {
            "ensure_rgb": false,
            "ensure_grayscale": false,
            "imagenet_mode": null,
            "input_scaling": 0.5,
            "pad_to_stride": 16,
            "resize_and_pad_to_target": true,
            "target_height": 1024,
            "target_width": 1024
        },
        "instance_cropping": {
            "center_on_part": "thorax",
            "crop_size": null,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 16,
                "output_stride": 2,
                "filters": 16,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": null,
            "centroid": {
                "anchor_part": "thorax",
                "sigma": 2.5,
                "output_stride": 2,
                "loss_weight": 1.0,
                "offset_refinement": false
            },
            "centered_instance": null,
            "multi_instance": null,
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        },
        "base_checkpoint": null
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -180.0,
            "rotation_max_angle": 180.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": false,
            "flip_horizontal": false
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": 200,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": 10,
        "min_val_batches_per_epoch": 10,
        "epochs": 2,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 20
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "240903_105332.centroid.n=100",
        "run_name_prefix": "",
        "run_name_suffix": "",
        "runs_folder": "/home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": true,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": true,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.3.3",
    "filename": "/tmp/tmpd1et2_6d/240903_105333_training_job.json"
}
INFO:sleap.nn.training:
2024-09-03 10:53:34.958393: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:53:34.962997: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:53:34.963711: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
INFO:sleap.nn.training:Auto-selected GPU 0 with 23692 MiB of free memory.
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training:  Splits: Training = 90 / Validation = 10.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
2024-09-03 10:53:35.790360: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-09-03 10:53:35.791013: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:53:35.791858: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:53:35.792617: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:53:36.063704: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:53:36.064484: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:53:36.065168: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:53:36.065830: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21787 MB memory:  -> device: 0, name: NVIDIA RTX A5000, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:sleap.nn.training:Loaded test example. [1.300s]
INFO:sleap.nn.training:  Input shape: (512, 512, 1)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training:  Backbone: UNet(stacks=1, filters=16, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=3, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training:  Max stride: 16
INFO:sleap.nn.training:  Parameters: 1,953,105
INFO:sleap.nn.training:  Heads: 
INFO:sleap.nn.training:    [0] = CentroidConfmapsHead(anchor_part='thorax', sigma=2.5, output_stride=2, loss_weight=1.0)
INFO:sleap.nn.training:  Outputs: 
INFO:sleap.nn.training:    [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 256, 256, 1), dtype=tf.float32, name=None), name='CentroidConfmapsHead/BiasAdd:0', description="created by layer 'CentroidConfmapsHead'")
INFO:sleap.nn.training:Training from scratch
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 90
INFO:sleap.nn.training:Validation set: n = 10
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training:  Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training:  Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=20)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.callbacks:Training controller subscribed to: tcp://127.0.0.1:9000 (topic: )
INFO:sleap.nn.training:  ZMQ controller subcribed to: tcp://127.0.0.1:9000
INFO:sleap.nn.callbacks:Progress reporter publishing on: tcp://127.0.0.1:9001 for: not_set
INFO:sleap.nn.training:  ZMQ progress reporter publish on: tcp://127.0.0.1:9001
INFO:sleap.nn.training:Created run path: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105332.centroid.n=100
INFO:sleap.nn.training:Setting up visualization...
INFO:sleap.nn.training:Finished trainer set up. [3.2s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
INFO:sleap.nn.training:Finished creating training datasets. [3.5s]
INFO:sleap.nn.training:Starting training loop...
Epoch 1/2
2024-09-03 10:53:43.671648: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0073s vs `on_train_batch_end` time: 0.0272s). Check your callbacks.
2024-09-03 10:53:58.686686: I tensorflow/stream_executor/cuda/cuda_blas.cc:1774] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
200/200 - 17s - loss: 2.1690e-04 - val_loss: 1.0114e-04 - lr: 1.0000e-04 - 17s/epoch - 84ms/step
Epoch 2/2
Polling: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105332.centroid.n=100/viz/validation.*.png
200/200 - 9s - loss: 6.7886e-05 - val_loss: 4.3949e-05 - lr: 1.0000e-04 - 9s/epoch - 45ms/step
Polling: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105332.centroid.n=100/viz/validation.*.png
INFO:sleap.nn.training:Finished training loop. [0.4 min]
INFO:sleap.nn.training:Deleting visualization directory: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105332.centroid.n=100/viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 34.9 FPS
INFO:sleap.nn.evals:Saved predictions: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105332.centroid.n=100/labels_pr.train.slp
INFO:sleap.nn.evals:Saved metrics: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105332.centroid.n=100/metrics.train.npz
INFO:sleap.nn.evals:OKS mAP: 0.737803
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 7.3 FPS
INFO:sleap.nn.evals:Saved predictions: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105332.centroid.n=100/labels_pr.val.slp
INFO:sleap.nn.evals:Saved metrics: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105332.centroid.n=100/metrics.val.npz
INFO:sleap.nn.evals:OKS mAP: 0.633989
INFO:sleap.nn.callbacks:Closing the reporter controller/context.
INFO:sleap.nn.callbacks:Closing the training controller socket/context.
Run Path: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105332.centroid.n=100
Finished training centroid.
Resetting monitor window.
Polling: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105416.centered_instance.n=100/viz/validation.*.png
Start training centered_instance...
['sleap-train', '/tmp/tmpnk568sb3/240903_105416_training_job.json', '/home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/courtship_labels.slp', '--zmq', '--save_viz']
INFO:sleap.nn.training:Versions:
SLEAP: 1.3.4
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Linux-5.15.0-78-generic-x86_64-with-debian-bookworm-sid
INFO:sleap.nn.training:Training labels file: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Training profile: /tmp/tmpnk568sb3/240903_105416_training_job.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
    "training_job_path": "/tmp/tmpnk568sb3/240903_105416_training_job.json",
    "labels_path": "/home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/courtship_labels.slp",
    "video_paths": [
        ""
    ],
    "val_labels": null,
    "test_labels": null,
    "base_checkpoint": null,
    "tensorboard": false,
    "save_viz": true,
    "zmq": true,
    "run_name": "",
    "prefix": "",
    "suffix": "",
    "cpu": false,
    "first_gpu": false,
    "last_gpu": false,
    "gpu": "auto"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
    "data": {
        "labels": {
            "training_labels": "/home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/courtship_labels.slp",
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": [
                66,
                96,
                58,
                4,
                65,
                61,
                85,
                28,
                31,
                35,
                90,
                60,
                16,
                27,
                55,
                13,
                9,
                80,
                87,
                6,
                75,
                70,
                95,
                56,
                34,
                82,
                69,
                76,
                89,
                18,
                39,
                77,
                51,
                33,
                24,
                32,
                81,
                7,
                53,
                20,
                36,
                67,
                54,
                12,
                10,
                19,
                30,
                86,
                44,
                71,
                0,
                73,
                40,
                83,
                2,
                14,
                38,
                74,
                94,
                68,
                98,
                17,
                49,
                57,
                84,
                78,
                72,
                26,
                93,
                92,
                5,
                3,
                47,
                1,
                88,
                37,
                45,
                99,
                46,
                63,
                62,
                23,
                8,
                42,
                50,
                48,
                91,
                64,
                79,
                43
            ],
            "validation_inds": [
                15,
                11,
                25,
                21,
                22,
                29,
                41,
                52,
                97,
                59
            ],
            "test_inds": null,
            "search_path_hints": [
                "",
                "",
                "",
                "",
                "",
                ""
            ],
            "skeletons": []
        },
        "preprocessing": {
            "ensure_rgb": false,
            "ensure_grayscale": false,
            "imagenet_mode": null,
            "input_scaling": 1.0,
            "pad_to_stride": 1,
            "resize_and_pad_to_target": true,
            "target_height": 1024,
            "target_width": 1024
        },
        "instance_cropping": {
            "center_on_part": "thorax",
            "crop_size": 144,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 16,
                "output_stride": 4,
                "filters": 24,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": null,
            "centroid": null,
            "centered_instance": {
                "anchor_part": "thorax",
                "part_names": [
                    "head",
                    "thorax",
                    "abdomen",
                    "wingL",
                    "wingR",
                    "forelegL4",
                    "forelegR4",
                    "midlegL4",
                    "midlegR4",
                    "hindlegL4",
                    "hindlegR4",
                    "eyeL",
                    "eyeR"
                ],
                "sigma": 2.5,
                "output_stride": 4,
                "loss_weight": 1.0,
                "offset_refinement": false
            },
            "multi_instance": null,
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        },
        "base_checkpoint": null
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -180.0,
            "rotation_max_angle": 180.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": false,
            "flip_horizontal": false
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": 200,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": 10,
        "min_val_batches_per_epoch": 10,
        "epochs": 2,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 10
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "240903_105416.centered_instance.n=100",
        "run_name_prefix": "",
        "run_name_suffix": "",
        "runs_folder": "/home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": true,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": true,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.3.3",
    "filename": "/tmp/tmpnk568sb3/240903_105416_training_job.json"
}
INFO:sleap.nn.training:
2024-09-03 10:54:18.223901: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:54:18.227782: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:54:18.228513: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
INFO:sleap.nn.training:Auto-selected GPU 0 with 23719 MiB of free memory.
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training:  Splits: Training = 90 / Validation = 10.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
2024-09-03 10:54:19.052567: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-09-03 10:54:19.053454: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:54:19.054528: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:54:19.055374: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:54:19.317241: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:54:19.317998: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:54:19.318680: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:54:19.319340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 21814 MB memory:  -> device: 0, name: NVIDIA RTX A5000, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:sleap.nn.training:Loaded test example. [1.644s]
INFO:sleap.nn.training:  Input shape: (144, 144, 1)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training:  Backbone: UNet(stacks=1, filters=24, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=2, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training:  Max stride: 16
INFO:sleap.nn.training:  Parameters: 4,311,445
INFO:sleap.nn.training:  Heads: 
INFO:sleap.nn.training:    [0] = CenteredInstanceConfmapsHead(part_names=['head', 'thorax', 'abdomen', 'wingL', 'wingR', 'forelegL4', 'forelegR4', 'midlegL4', 'midlegR4', 'hindlegL4', 'hindlegR4', 'eyeL', 'eyeR'], anchor_part='thorax', sigma=2.5, output_stride=4, loss_weight=1.0)
INFO:sleap.nn.training:  Outputs: 
INFO:sleap.nn.training:    [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 36, 36, 13), dtype=tf.float32, name=None), name='CenteredInstanceConfmapsHead/BiasAdd:0', description="created by layer 'CenteredInstanceConfmapsHead'")
INFO:sleap.nn.training:Training from scratch
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 90
INFO:sleap.nn.training:Validation set: n = 10
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training:  Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training:  Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=10)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.callbacks:Training controller subscribed to: tcp://127.0.0.1:9000 (topic: )
INFO:sleap.nn.training:  ZMQ controller subcribed to: tcp://127.0.0.1:9000
INFO:sleap.nn.callbacks:Progress reporter publishing on: tcp://127.0.0.1:9001 for: not_set
INFO:sleap.nn.training:  ZMQ progress reporter publish on: tcp://127.0.0.1:9001
INFO:sleap.nn.training:Created run path: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105416.centered_instance.n=100
INFO:sleap.nn.training:Setting up visualization...
INFO:sleap.nn.training:Finished trainer set up. [3.0s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
INFO:sleap.nn.training:Finished creating training datasets. [4.2s]
INFO:sleap.nn.training:Starting training loop...
Epoch 1/2
2024-09-03 10:54:27.519675: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0093s vs `on_train_batch_end` time: 0.0115s). Check your callbacks.
2024-09-03 10:54:35.713655: I tensorflow/stream_executor/cuda/cuda_blas.cc:1774] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
200/200 - 10s - loss: 0.0110 - head: 0.0078 - thorax: 0.0070 - abdomen: 0.0109 - wingL: 0.0121 - wingR: 0.0125 - forelegL4: 0.0114 - forelegR4: 0.0109 - midlegL4: 0.0130 - midlegR4: 0.0131 - hindlegL4: 0.0131 - hindlegR4: 0.0132 - eyeL: 0.0093 - eyeR: 0.0092 - val_loss: 0.0088 - val_head: 0.0036 - val_thorax: 0.0037 - val_abdomen: 0.0073 - val_wingL: 0.0085 - val_wingR: 0.0105 - val_forelegL4: 0.0098 - val_forelegR4: 0.0095 - val_midlegL4: 0.0120 - val_midlegR4: 0.0129 - val_hindlegL4: 0.0123 - val_hindlegR4: 0.0124 - val_eyeL: 0.0057 - val_eyeR: 0.0068 - lr: 1.0000e-04 - 10s/epoch - 51ms/step
Epoch 2/2
Polling: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105332.centroid.n=100/viz/validation.*.png
Polling: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105416.centered_instance.n=100/viz/validation.*.png
200/200 - 6s - loss: 0.0073 - head: 0.0023 - thorax: 0.0033 - abdomen: 0.0060 - wingL: 0.0069 - wingR: 0.0082 - forelegL4: 0.0083 - forelegR4: 0.0082 - midlegL4: 0.0107 - midlegR4: 0.0114 - hindlegL4: 0.0117 - hindlegR4: 0.0117 - eyeL: 0.0033 - eyeR: 0.0033 - val_loss: 0.0064 - val_head: 0.0020 - val_thorax: 0.0030 - val_abdomen: 0.0051 - val_wingL: 0.0065 - val_wingR: 0.0072 - val_forelegL4: 0.0065 - val_forelegR4: 0.0070 - val_midlegL4: 0.0091 - val_midlegR4: 0.0105 - val_hindlegL4: 0.0105 - val_hindlegR4: 0.0105 - val_eyeL: 0.0028 - val_eyeR: 0.0030 - lr: 1.0000e-04 - 6s/epoch - 31ms/step
INFO:sleap.nn.training:Finished training loop. [0.3 min]
INFO:sleap.nn.training:Deleting visualization directory: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105416.centered_instance.n=100/viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?Polling: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105332.centroid.n=100/viz/validation.*.png
Polling: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105416.centered_instance.n=100/viz/validation.*.png
Predicting... ━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  18% ETA: 0:00:01 210.8 FPS2024-09-03 10:54:44.962420: W tensorflow/core/data/root_dataset.cc:163] Optimization loop failed: CANCELLED: Operation was cancelled
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 52.8 FPS
INFO:sleap.nn.evals:Saved predictions: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105416.centered_instance.n=100/labels_pr.train.slp
INFO:sleap.nn.evals:Saved metrics: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105416.centered_instance.n=100/metrics.train.npz
INFO:sleap.nn.evals:OKS mAP: 0.013254
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 11.8 FPS
INFO:sleap.nn.evals:Saved predictions: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105416.centered_instance.n=100/labels_pr.val.slp
INFO:sleap.nn.evals:Saved metrics: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105416.centered_instance.n=100/metrics.val.npz
INFO:sleap.nn.evals:OKS mAP: 0.010202
INFO:sleap.nn.callbacks:Closing the reporter controller/context.
INFO:sleap.nn.callbacks:Closing the training controller socket/context.
Run Path: /home/talmolab/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/drosophila-melanogaster-courtship/models/240903_105416.centered_instance.n=100
Finished training centered_instance.
Terminal: Opening GUI 


Nothing.






 sleap-label

(sleap_1.3) talmolab@talmolab-01-ubuntu:~$ sleap-label
Saving config: /home/talmolab/.sleap/1.3.4/preferences.yaml
2024-09-03 10:45:48.326588: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:45:48.330951: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-09-03 10:45:48.331731: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

Software versions:
SLEAP: 1.3.4
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Linux-5.15.0-78-generic-x86_64-with-debian-bookworm-sid

Happy SLEAPing! :)





  mm create sleap

talmolab@talmolab-01-ubuntu:~$ mm create -y -n -c conda-forge -c nvidia -c sleap/label/dev -c sleap -c anaconda sleap=1.3.4
nvidia/noarch (check zst)                           Checked  0.1s
nvidia/linux-64 (check zst)                         Checked  0.1s
anaconda/noarch (check zst)                        Checked  0.1s
anaconda/linux-64 (check zst)                       Checked  0.0s
sleap/label/dev/noarch (check zst)                  Checked  0.2s
sleap/label/dev/linux-64 (check zst)                Checked  0.2s
sleap/linux-64 (check zst)                          Checked  0.2s
sleap/noarch (check zst)                            Checked  0.2s
nvidia/noarch                                       14.7kB @ 291.8kB/s  0.1s
nvidia/linux-64                                    210.3kB @   2.7MB/s  0.1s
sleap/label/dev/noarch                             116.0 B @ 792.0 B/s  0.1s
anaconda/linux-64                                    3.0MB @  17.7MB/s  0.1s
sleap/label/dev/linux-64                             1.6kB @   8.2kB/s  0.2s
conda-forge/noarch                                  16.3MB @  57.0MB/s  0.2s
anaconda/noarch                                    402.7kB @   1.3MB/s  0.1s
sleap/linux-64                                       2.4kB @   7.4kB/s  0.3s
sleap/noarch                                       116.0 B @ 330.0 B/s  0.2s
conda-forge/linux-64                                37.5MB @  51.7MB/s  0.7s
error    libmamba Could not solve for environment specs
    The following package could not be installed
    └─ conda-forge does not exist (perhaps a typo or a missing channel).
critical libmamba Could not solve for environment specs
talmolab@talmolab-01-ubuntu:~$ mm create -y -n -c conda-forge -c nvidia -c sleap/label/dev -c sleap -c anaconda sleap=1.3.4
nvidia/linux-64                                             Using cache
nvidia/noarch                                               Using cache
anaconda/linux-64                                           Using cache
anaconda/noarch                                             Using cache
conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
sleap/linux-64                                                No change
sleap/label/dev/linux-64                                      No change
sleap/label/dev/noarch                                        No change
sleap/noarch                                                  No change
error    libmamba Could not solve for environment specs
    The following package could not be installed
    └─ conda-forge does not exist (perhaps a typo or a missing channel).
critical libmamba Could not solve for environment specs
talmolab@talmolab-01-ubuntu:~$ mm create -y -n sleap_1.3.4 -c conda-forge -c nvidia -c sleap/label/dev -c sleap -c anaconda sleap=1.3.4
conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
nvidia/linux-64                                             Using cache
nvidia/noarch                                               Using cache
anaconda/linux-64                                           Using cache
anaconda/noarch                                             Using cache
sleap/label/dev/linux-64                                      No change
sleap/label/dev/noarch                                        No change
sleap/noarch                                                  No change
sleap/linux-64                                                No change

Transaction

  Prefix: /home/talmolab/micromamba/envs/sleap_1.3.4

  Updating specs:

   - sleap=1.3.4


  Package                    Version  Build                   Channel               Size
──────────────────────────────────────────────────────────────────────────────────────────
  Install:
──────────────────────────────────────────────────────────────────────────────────────────

  + cuda-nvcc                11.3.58  h2467b9f_0              nvidia              Cached
  + python_abi                   3.7  4_cp37m                 conda-forge         Cached
  + _libgcc_mutex                0.1  conda_forge             conda-forge         Cached
  + ld_impl_linux-64            2.40  hf3520f5_7              conda-forge          708kB
  + ca-certificates        2024.8.30  hbcca054_0              conda-forge          159kB
  + libgomp                   14.1.0  h77fa898_1              conda-forge          460kB
  + _openmp_mutex                4.5  2_gnu                   conda-forge         Cached
  + libgcc                    14.1.0  h77fa898_1              conda-forge          846kB
  + libbrotlicommon            1.1.0  hb9d3cd8_2              conda-forge           69kB
  + libgfortran5              14.1.0  hc5f4f2c_1              conda-forge            1MB
  + libstdcxx                 14.1.0  hc0a3c3a_1              conda-forge            4MB
  + libgcc-ng                 14.1.0  h69a702a_1              conda-forge           52kB
  + libbrotlienc               1.1.0  hb9d3cd8_2              conda-forge          282kB
  + libbrotlidec               1.1.0  hb9d3cd8_2              conda-forge           33kB
  + libgfortran               14.1.0  h69a702a_1              conda-forge           52kB
  + libstdcxx-ng              14.1.0  h4852527_1              conda-forge           52kB
  + libev                       4.33  hd590300_2              conda-forge          113kB
  + c-ares                    1.33.1  heb4867d_0              conda-forge          183kB
  + libogg                     1.3.5  h4ab18f5_0              conda-forge          206kB
  + yaml                       0.2.5  h7f98852_2              conda-forge         Cached
  + libsodium                 1.0.18  h36c2ea0_1              conda-forge         Cached
  + xorg-renderproto          0.11.1  h7f98852_1002           conda-forge         Cached
  + xorg-xproto               7.0.31  h7f98852_1007           conda-forge         Cached
  + xorg-kbproto               1.0.7  h7f98852_1002           conda-forge         Cached
  + xorg-xextproto             7.3.0  h0b41bf4_1003           conda-forge         Cached
  + libuuid                   2.38.1  h0b41bf4_0              conda-forge         Cached
  + xorg-libice                1.1.1  hd590300_0              conda-forge         Cached
  + x264                  1!161.3030  h7f98852_1              conda-forge         Cached
  + lame                       3.100  h166bdaf_1003           conda-forge         Cached
  + nettle                       3.6  he412f7d_0              conda-forge         Cached
  + bzip2                      1.0.8  h4bc722e_7              conda-forge          253kB
  + libopus                    1.3.1  h7f98852_1              conda-forge         Cached
  + alsa-lib                 1.2.3.2  h166bdaf_0              conda-forge         Cached
  + libgettextpo              0.22.5  he02047a_3              conda-forge          171kB
  + gettext-tools             0.22.5  he02047a_3              conda-forge            3MB
  + libwebp-base               1.4.0  hd590300_0              conda-forge          439kB
  + libdeflate                  1.14  h166bdaf_0              conda-forge         Cached
  + keyutils                   1.6.1  h166bdaf_0              conda-forge         Cached
  + xorg-libxdmcp              1.1.3  h7f98852_0              conda-forge         Cached
  + xorg-libxau               1.0.11  hd590300_0              conda-forge         Cached
  + pthread-stubs                0.4  h36c2ea0_1001           conda-forge         Cached
  + libiconv                    1.17  hd590300_2              conda-forge          706kB
  + jpeg                          9e  h0b41bf4_3              conda-forge         Cached
  + libexpat                   2.6.2  h59595ed_0              conda-forge           74kB
  + xz                         5.2.6  h166bdaf_0              conda-forge         Cached
  + openssl                   1.1.1w  hd590300_0              conda-forge         Cached
  + ncurses                      6.5  he02047a_1              conda-forge          889kB
  + libzlib                   1.2.13  h4ab18f5_6              conda-forge           62kB
  + libnsl                     2.0.1  hd590300_0              conda-forge           33kB
  + libffi                     3.4.2  h7f98852_5              conda-forge         Cached
  + brotli-bin                 1.1.0  hb9d3cd8_2              conda-forge           19kB
  + libgfortran-ng            14.1.0  h69a702a_1              conda-forge           52kB
  + geos                      3.11.0  h27087fc_0              conda-forge         Cached
  + pixman                    0.43.2  h59595ed_0              conda-forge          387kB
  + gmp                        6.3.0  hac33072_2              conda-forge          460kB
  + libasprintf               0.22.5  he8f35ee_3              conda-forge           43kB
  + graphite2                 1.3.13  h59595ed_1003           conda-forge           97kB
  + lerc                       4.0.0  h27087fc_0              conda-forge         Cached
  + nspr                        4.35  h27087fc_0              conda-forge         Cached
  + icu                         68.2  h9c3ff4c_0              conda-forge         Cached
  + cudatoolkit               11.3.1  hb98b00a_13             conda-forge          633MB
  + libvorbis                  1.3.7  h9c3ff4c_0              conda-forge         Cached
  + zeromq                     4.3.5  h59595ed_1              conda-forge          343kB
  + xorg-libsm                 1.2.4  h7391055_0              conda-forge         Cached
  + gnutls                    3.6.13  h85f3911_1              conda-forge         Cached
  + libgettextpo-devel        0.22.5  he02047a_3              conda-forge           37kB
  + libxcb                      1.13  h7f98852_1004           conda-forge         Cached
  + jasper                   1.900.1  h07fcdf6_1006           conda-forge         Cached
  + expat                      2.6.2  h59595ed_0              conda-forge          138kB
  + mysql-common              8.0.32  h14678bc_0              conda-forge         Cached
  + libevent                  2.1.10  h9b69904_4              conda-forge         Cached
  + libedit             3.1.20191231  he28a2e2_2              conda-forge         Cached
  + readline                     8.2  h8228510_1              conda-forge         Cached
  + libssh2                   1.10.0  haa6b8db_3              conda-forge         Cached
  + libnghttp2                1.51.0  hdcd2b5c_0              conda-forge         Cached
  + zstd                       1.5.6  ha6fb4c9_0              conda-forge          555kB
  + libllvm11                 11.1.0  he0ac6c6_5              conda-forge         Cached
  + libpng                    1.6.43  h2797004_0              conda-forge          288kB
  + pcre2                      10.43  hcad00b1_0              conda-forge          951kB
  + zlib                      1.2.13  h4ab18f5_6              conda-forge           93kB
  + tk                        8.6.13  noxft_h4845f30_101      conda-forge            3MB
  + libsqlite                 3.46.0  hde9e2c9_0              conda-forge          865kB
  + brotli                     1.1.0  hb9d3cd8_2              conda-forge           19kB
  + fftw                      3.3.10  nompi_hf1063bd_110      conda-forge            2MB
  + libopenblas               0.3.27  pthreads_hac2b453_1     conda-forge            6MB
  + libasprintf-devel         0.22.5  he8f35ee_3              conda-forge           34kB
  + cudnn                   8.2.1.32  h86fa8c9_0              conda-forge         Cached
  + xorg-libx11                1.8.4  h0b41bf4_0              conda-forge         Cached
  + krb5                      1.19.3  h3790be6_0              conda-forge         Cached
  + libtiff                    4.4.0  h82bc61c_5              conda-forge         Cached
  + mysql-libs                8.0.32  h54cf53e_0              conda-forge         Cached
  + libclang                  11.1.0  default_ha53f305_1      conda-forge         Cached
  + freetype                  2.12.1  h267a509_2              conda-forge         Cached
  + libglib                   2.80.2  hf974151_0              conda-forge            4MB
  + openh264                   2.1.1  h780b84a_0              conda-forge         Cached
  + libxml2                   2.9.12  h72842e0_0              conda-forge         Cached
  + libprotobuf               3.16.0  h780b84a_0              conda-forge         Cached
  + nss                        3.100  hca3bf56_0              conda-forge            2MB
  + sqlite                    3.46.0  h6d4b2fc_0              conda-forge          860kB
  + openblas                  0.3.27  pthreads_h9eca1d5_1     conda-forge            6MB
  + libblas                    3.9.0  23_linux64_openblas     conda-forge           15kB
  + gettext                   0.22.5  he02047a_3              conda-forge          479kB
  + xorg-libxext               1.3.4  h0b41bf4_2              conda-forge         Cached
  + xorg-libxrender           0.9.10  h7f98852_1003           conda-forge         Cached
  + libcurl                   7.86.0  h7bff187_1              conda-forge         Cached
  + libpq                       13.8  hd77ab85_0              conda-forge         Cached
  + openjpeg                   2.5.0  h7d73246_1              conda-forge         Cached
  + lcms2                       2.14  h6ed2654_0              conda-forge         Cached
  + fontconfig                2.14.2  h14ed4e7_0              conda-forge         Cached
  + dbus                      1.13.6  h5008d03_3              conda-forge         Cached
  + ffmpeg                     4.3.2  h37c90e5_3              conda-forge         Cached
  + libxslt                   1.1.33  h15afd5d_2              conda-forge         Cached
  + libxkbcommon               1.0.3  he3ba5ed_0              conda-forge         Cached
  + python                    3.7.12  hb7a2778_100_cpython    conda-forge         Cached
  + blas                         1.1  openblas                conda-forge            1kB
  + libcblas                   3.9.0  23_linux64_openblas     conda-forge           15kB
  + liblapack                  3.9.0  23_linux64_openblas     conda-forge           15kB
  + gstreamer                 1.18.5  h9f60fe5_3              conda-forge         Cached
  + hdf5                      1.10.6  nompi_h6a2412b_1114     conda-forge         Cached
  + cairo                     1.16.0  h6cf1ce9_1008           conda-forge         Cached
  + setuptools                59.8.0  py37h89c1867_1          conda-forge         Cached
  + liblapacke                 3.9.0  23_linux64_openblas     conda-forge           15kB
  + gst-plugins-base          1.18.5  hf529b03_3              conda-forge         Cached
  + harfbuzz                   2.9.1  h83ec7ef_1              conda-forge         Cached
  + qt                        5.12.9  hda022c4_4              conda-forge         Cached
  + libopencv                  4.5.3  py37h25009ff_1          conda-forge         Cached
  + wheel                     0.42.0  pyhd8ed1ab_0            conda-forge           58kB
  + pip                         24.0  pyhd8ed1ab_0            conda-forge            1MB
  + locket                     1.0.0  pyhd8ed1ab_0            conda-forge         Cached
  + fsspec                  2023.1.0  pyhd8ed1ab_0            conda-forge         Cached
  + zipp                      3.15.0  pyhd8ed1ab_0            conda-forge           17kB
  + toolz                     0.12.1  pyhd8ed1ab_0            conda-forge           52kB
  + cloudpickle                2.2.1  pyhd8ed1ab_0            conda-forge         Cached
  + threadpoolctl              3.1.0  pyh8a188c0_0            conda-forge         Cached
  + joblib                     1.3.2  pyhd8ed1ab_0            conda-forge         Cached
  + mdurl                      0.1.2  pyhd8ed1ab_0            conda-forge           15kB
  + six                       1.16.0  pyh6c4a22f_0            conda-forge         Cached
  + pygments                  2.17.2  pyhd8ed1ab_0            conda-forge          860kB
  + jsonpickle                   1.2  py_0                    conda-forge         Cached
  + attrs                     21.4.0  pyhd8ed1ab_0            conda-forge         Cached
  + munkres                    1.1.4  pyh9f0ad1d_0            conda-forge         Cached
  + typing_extensions          4.7.1  pyha770c72_0            conda-forge         Cached
  + pyparsing                  3.1.4  pyhd8ed1ab_0            conda-forge           90kB
  + packaging                   23.2  pyhd8ed1ab_0            conda-forge         Cached
  + cycler                    0.11.0  pyhd8ed1ab_0            conda-forge         Cached
  + certifi                2024.8.30  pyhd8ed1ab_0            conda-forge          164kB
  + pytz                      2024.1  pyhd8ed1ab_0            conda-forge          189kB
  + cached_property            1.5.2  pyha770c72_1            conda-forge         Cached
  + jsmin                      3.0.1  pyhd8ed1ab_0            conda-forge         Cached
  + partd                      1.4.1  pyhd8ed1ab_0            conda-forge         Cached
  + python-dateutil            2.9.0  pyhd8ed1ab_0            conda-forge          223kB
  + cattrs                     1.1.1  pyhd8ed1ab_0            conda-forge         Cached
  + markdown-it-py             2.2.0  pyhd8ed1ab_0            conda-forge         Cached
  + typing-extensions          4.7.1  hd8ed1ab_0              conda-forge         Cached
  + qtpy                       2.4.1  pyhd8ed1ab_0            conda-forge           62kB
  + cached-property            1.5.2  hd8ed1ab_1              conda-forge         Cached
  + rich                      13.7.1  pyhd8ed1ab_0            conda-forge          184kB
  + pyzmq                     24.0.1  py37h0c0c2a8_0          conda-forge         Cached
  + pyyaml                       6.0  py37h540881e_4          conda-forge         Cached
  + python-rapidjson             1.9  py37hd23a5d3_0          conda-forge         Cached
  + pyside2                   5.13.2  py37hfa98aef_7          conda-forge         Cached
  + psutil                     5.9.3  py37h540881e_0          conda-forge         Cached
  + unicodedata2              14.0.0  py37h540881e_1          conda-forge         Cached
  + pillow                     9.2.0  py37h850a105_2          conda-forge         Cached
  + numpy                     1.21.6  py37h976b520_0          conda-forge         Cached
  + cytoolz                   0.12.0  py37h540881e_0          conda-forge         Cached
  + protobuf                  3.16.0  py37hcd2ae1e_0          conda-forge         Cached
  + importlib-metadata        4.11.4  py37h89c1867_0          conda-forge           34kB
  + kiwisolver                 1.4.4  py37h7cecad7_0          conda-forge         Cached
  + fonttools                 4.38.0  py37h540881e_0          conda-forge         Cached
  + imagecodecs-lite       2019.12.3  py37hc105733_5          conda-forge         Cached
  + pywavelets                 1.3.0  py37hda87dfa_1          conda-forge         Cached
  + shapely                    1.8.5  py37ha4e3bd1_0          conda-forge         Cached
  + pandas                     1.3.5  py37he8f5f7f_0          conda-forge         Cached
  + py-opencv                  4.5.3  py37h6531663_1          conda-forge         Cached
  + h5py                       3.1.0  nompi_py37h1e651dc_100  conda-forge         Cached
  + matplotlib-base            3.5.3  py37hf395dca_2          conda-forge         Cached
  + opencv                     4.5.3  py37h89c1867_1          conda-forge         Cached
  + dask-core               2022.2.0  pyhd8ed1ab_0            conda-forge         Cached
  + patsy                      0.5.6  pyhd8ed1ab_0            conda-forge          187kB
  + imageio                   2.35.1  pyh12aca89_0            conda-forge          293kB
  + tifffile                2020.6.3  py_0                    conda-forge         Cached
  + scipy                      1.7.3  py37hf838250_2          anaconda              23MB
  + tensorflow                 2.7.0  py37hb93dfd8_4          sleap/label/dev      487MB
  + statsmodels               0.13.2  py37hda87dfa_0          conda-forge         Cached
  + scikit-learn                 1.0  py37hf0f1638_1          conda-forge         Cached
  + seaborn-base              0.12.2  pyhd8ed1ab_0            conda-forge         Cached
  + pykalman                   0.9.7  pyhd8ed1ab_0            conda-forge          232kB
  + networkx                     2.7  pyhd8ed1ab_0            conda-forge         Cached
  + scikit-video              1.1.11  pyh24bf2e0_0            conda-forge         Cached
  + tensorflow-hub            0.13.0  pyh56297ac_0            conda-forge         Cached
  + seaborn                   0.12.2  hd8ed1ab_0              conda-forge         Cached
  + scikit-image              0.19.2  py37he8f5f7f_0          conda-forge         Cached
  + imgaug                     0.4.0  pyhd8ed1ab_1            conda-forge         Cached
  + sleap                      1.3.4  py37_2                  sleap/label/dev        5MB

  Summary:

  Install: 195 packages

  Total download: 1GB





Windows


 GUI Training

image

image


 Terminal: Training

Resetting monitor window.
Polling: C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\models\240831_110334.multi_instance.n=101\viz\validation.*.png
Start training multi_instance...
['sleap-train', 'C:\\Users\\Liezl\\AppData\\Local\\Temp\\tmp65eu9h6c\\240831_110334_training_job.json', 'C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/courtship_labels.slp', '--zmq', '--save_viz']
INFO:sleap.nn.training:Versions:
SLEAP: 1.3.4
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Windows-10-10.0.22621-SP0
INFO:sleap.nn.training:Training labels file: C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Training profile: C:\Users\Liezl\AppData\Local\Temp\tmp65eu9h6c\240831_110334_training_job.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
    "training_job_path": "C:\\Users\\Liezl\\AppData\\Local\\Temp\\tmp65eu9h6c\\240831_110334_training_job.json",
    "labels_path": "C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/courtship_labels.slp",
    "video_paths": [
        ""
    ],
    "val_labels": null,
    "test_labels": null,
    "base_checkpoint": null,
    "tensorboard": false,
    "save_viz": true,
    "zmq": true,
    "run_name": "",
    "prefix": "",
    "suffix": "",
    "cpu": false,
    "first_gpu": false,
    "last_gpu": false,
    "gpu": "auto"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
    "data": {
        "labels": {
            "training_labels": "D:\\Users\\Liezl\\ProjectsData\\sleap-estimates-animal-poses\\datasets\\drosophila-melanogaster-courtship\\courtship_labels.slp",
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": [
                7,
                50,
                29,
                42,
                73,
                67,
                89,
                46,
                17,
                47,
                11,
                65,
                16,
                69,
                13,
                72,
                93,
                8,
                23,
                6,
                9,
                5,
                97,
                53,
                91,
                19,
                100,
                71,
                31,
                94,
                36,
                85,
                18,
                83,
                3,
                82,
                21,
                74,
                1,
                86,
                81,
                64,
                51,
                30,
                54,
                52,
                58,
                95,
                92,
                62,
                48,
                76,
                27,
                45,
                15,
                57,
                79,
                4,
                56,
                98,
                84,
                38,
                78,
                34,
                90,
                63,
                87,
                25,
                44,
                32,
                2,
                28,
                55,
                22,
                61,
                66,
                33,
                41,
                14,
                88,
                26,
                68,
                12,
                39,
                60,
                99,
                24,
                77,
                43,
                10,
                80
            ],
            "validation_inds": [
                37,
                0,
                96,
                40,
                49,
                70,
                75,
                20,
                35,
                59
            ],
            "test_inds": null,
            "search_path_hints": [
                ""
            ],
            "skeletons": []
        },
        "preprocessing": {
            "ensure_rgb": false,
            "ensure_grayscale": false,
            "imagenet_mode": null,
            "input_scaling": 1.0,
            "pad_to_stride": 32,
            "resize_and_pad_to_target": true,
            "target_height": 1024,
            "target_width": 1024
        },
        "instance_cropping": {
            "center_on_part": null,
            "crop_size": null,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 32,
                "output_stride": 4,
                "filters": 16,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": null,
            "centroid": null,
            "centered_instance": null,
            "multi_instance": {
                "confmaps": {
                    "part_names": [
                        "head",
                        "thorax",
                        "abdomen",
                        "wingL",
                        "wingR",
                        "forelegL4",
                        "forelegR4",
                        "midlegL4",
                        "midlegR4",
                        "hindlegL4",
                        "hindlegR4",
                        "eyeL",
                        "eyeR"
                    ],
                    "sigma": 2.5,
                    "output_stride": 4,
                    "loss_weight": 1.0,
                    "offset_refinement": false
                },
                "pafs": {
                    "edges": [
                        [
                            "thorax",
                            "head"
                        ],
                        [
                            "thorax",
                            "abdomen"
                        ],
                        [
                            "thorax",
                            "wingL"
                        ],
                        [
                            "thorax",
                            "wingR"
                        ],
                        [
                            "thorax",
                            "forelegL4"
                        ],
                        [
                            "thorax",
                            "forelegR4"
                        ],
                        [
                            "thorax",
                            "midlegL4"
                        ],
                        [
                            "thorax",
                            "midlegR4"
                        ],
                        [
                            "thorax",
                            "hindlegL4"
                        ],
                        [
                            "thorax",
                            "hindlegR4"
                        ],
                        [
                            "head",
                            "eyeL"
                        ],
                        [
                            "head",
                            "eyeR"
                        ]
                    ],
                    "sigma": 75.0,
                    "output_stride": 8,
                    "loss_weight": 1.0
                }
            },
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        },
        "base_checkpoint": null
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -180.0,
            "rotation_max_angle": 180.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": false,
            "flip_horizontal": false
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": 200,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": 10,
        "min_val_batches_per_epoch": 10,
        "epochs": 2,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 10
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "240831_110334.multi_instance.n=101",
        "run_name_prefix": "",
        "run_name_suffix": "",
        "runs_folder": "C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\\models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": true,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": true,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.4.1a2",
    "filename": "C:\\Users\\Liezl\\AppData\\Local\\Temp\\tmp65eu9h6c\\240831_110334_training_job.json"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Auto-selected GPU 0 with 11675 MiB of free memory.
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training:  Splits: Training = 91 / Validation = 10.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
2024-08-31 11:03:41.866195: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-08-31 11:03:42.671549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9599 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:02:00.0, compute capability: 8.6
INFO:sleap.nn.training:Loaded test example. [2.209s]
INFO:sleap.nn.training:  Input shape: (1024, 1024, 1)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training:  Backbone: UNet(stacks=1, filters=16, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=5, middle_block=True, up_blocks=3, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training:  Max stride: 32
INFO:sleap.nn.training:  Parameters: 7,819,861
INFO:sleap.nn.training:  Heads:
INFO:sleap.nn.training:    [0] = MultiInstanceConfmapsHead(part_names=['head', 'thorax', 'abdomen', 'wingL', 'wingR', 'forelegL4', 'forelegR4', 'midlegL4', 'midlegR4', 'hindlegL4', 'hindlegR4', 'eyeL', 'eyeR'], sigma=2.5, output_stride=4, loss_weight=1.0)
INFO:sleap.nn.training:    [1] = PartAffinityFieldsHead(edges=[('thorax', 'head'), ('thorax', 'abdomen'), ('thorax', 'wingL'), ('thorax', 'wingR'), ('thorax', 'forelegL4'), ('thorax', 'forelegR4'), ('thorax', 'midlegL4'), ('thorax', 'midlegR4'), ('thorax', 'hindlegL4'), ('thorax', 'hindlegR4'), ('head', 'eyeL'), ('head', 'eyeR')], sigma=75.0, output_stride=8, loss_weight=1.0)
INFO:sleap.nn.training:  Outputs:
INFO:sleap.nn.training:    [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 256, 256, 13), dtype=tf.float32, name=None), name='MultiInstanceConfmapsHead/BiasAdd:0', description="created by layer 'MultiInstanceConfmapsHead'")
INFO:sleap.nn.training:    [1] = KerasTensor(type_spec=TensorSpec(shape=(None, 128, 128, 24), dtype=tf.float32, name=None), name='PartAffinityFieldsHead/BiasAdd:0', description="created by layer 'PartAffinityFieldsHead'")
INFO:sleap.nn.training:Training from scratch
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 91
INFO:sleap.nn.training:Validation set: n = 10
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training:  Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training:  Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=10)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.callbacks:Training controller subscribed to: tcp://127.0.0.1:9000 (topic: )
INFO:sleap.nn.training:  ZMQ controller subcribed to: tcp://127.0.0.1:9000
INFO:sleap.nn.callbacks:Progress reporter publishing on: tcp://127.0.0.1:9001 for: not_set
INFO:sleap.nn.training:  ZMQ progress reporter publish on: tcp://127.0.0.1:9001
INFO:sleap.nn.training:Created run path: C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\models\240831_110334.multi_instance.n=101
INFO:sleap.nn.training:Setting up visualization...
INFO:sleap.nn.training:Finished trainer set up. [5.1s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
INFO:sleap.nn.training:Finished creating training datasets. [7.2s]
INFO:sleap.nn.training:Starting training loop...
Epoch 1/2
2024-08-31 11:03:57.202413: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0176s vs `on_train_batch_end` time: 0.1818s). Check your callbacks.
2024-08-31 11:05:17.899812: I tensorflow/stream_executor/cuda/cuda_blas.cc:1774] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
2024-08-31 11:05:20.997320: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2024-08-31 11:05:21.326703: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2024-08-31 11:05:21.392040: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.27GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
200/200 - 90s - loss: 0.0012 - MultiInstanceConfmapsHead_loss: 5.1135e-04 - PartAffinityFieldsHead_loss: 7.3340e-04 - val_loss: 0.0011 - val_MultiInstanceConfmapsHead_loss: 4.4563e-04 - val_PartAffinityFieldsHead_loss: 6.5197e-04 - lr: 1.0000e-04 - 90s/epoch - 450ms/step
Epoch 2/2
Polling: C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\models\240831_110334.multi_instance.n=101\viz\validation.*.png
200/200 - 58s - loss: 8.8963e-04 - MultiInstanceConfmapsHead_loss: 3.6029e-04 - PartAffinityFieldsHead_loss: 5.2933e-04 - val_loss: 6.1292e-04 - val_MultiInstanceConfmapsHead_loss: 2.5777e-04 - val_PartAffinityFieldsHead_loss: 3.5515e-04 - lr: 1.0000e-04 - 58s/epoch - 291ms/step
Polling: C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\models\240831_110334.multi_instance.n=101\viz\validation.*.png
INFO:sleap.nn.training:Finished training loop. [2.5 min]
INFO:sleap.nn.training:Deleting visualization directory: C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\models\240831_110334.multi_instance.n=101\viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
Predicting... ---------------------------------------- 100% ETA: 0:00:00 0.6 FPS
INFO:sleap.nn.evals:Saved predictions: C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\models\240831_110334.multi_instance.n=101\labels_pr.train.slp
INFO:sleap.nn.evals:Saved metrics: C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\models\240831_110334.multi_instance.n=101\metrics.train.npz
INFO:sleap.nn.evals:OKS mAP: 0.000000
Predicting... ---------------------------------------- 100% ETA: 0:00:00 1.2 FPS
INFO:sleap.nn.evals:Saved predictions: C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\models\240831_110334.multi_instance.n=101\labels_pr.val.slp
INFO:sleap.nn.evals:Saved metrics: C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\models\240831_110334.multi_instance.n=101\metrics.val.npz
INFO:sleap.nn.evals:OKS mAP: 0.000000
INFO:sleap.nn.callbacks:Closing the reporter controller/context.
INFO:sleap.nn.callbacks:Closing the training controller socket/context.
Run Path: C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\models\240831_110334.multi_instance.n=101
Finished training multi_instance.
Command line call:
sleap-track C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/courtship_labels.slp --only-suggested-frames -m C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\models\240831_110334.multi_instance.n=101 -o C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\predictions\courtship_labels.slp.240831_110650.predictions.slp --verbosity json --no-empty-frames

Started inference at: 2024-08-31 11:06:56.316214
Args:
{
    'data_path': 'C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/courtship_labels.slp',
    'models': [
        'C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\\models\\240831_110334.multi_instance.n=101'
    ],
    'frames': '',
    'only_labeled_frames': False,
    'only_suggested_frames': True,
    'output': 'C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\\predictions\\courtship_labels.slp.240831_110650.predictions.slp',
    'no_empty_frames': True,
    'verbosity': 'json',
    'video.dataset': None,
    'video.input_format': 'channels_last',
    'video.index': '',
    'cpu': False,
    'first_gpu': False,
    'last_gpu': False,
    'gpu': 'auto',
    'max_edge_length_ratio': 0.25,
    'dist_penalty_weight': 1.0,
    'batch_size': 4,
2024-08-31 11:06:57.480056: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
    'open_in_gui': False,
    'peak_threshold': 0.2,
    'max_instances': None,
    'tracking.tracker': None,
    'tracking.max_tracking': None,
    'tracking.max_tracks': None,
    'tracking.target_instance_count': None,
    'tracking.pre_cull_to_target': None,
2024-08-31 11:06:57.911416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9599 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3060, pci bus id: 0000:02:00.0, compute capability: 8.6
    'tracking.pre_cull_iou_threshold': None,
    'tracking.post_connect_single_breaks': None,
    'tracking.clean_instance_count': None,
    'tracking.clean_iou_threshold': None,
    'tracking.similarity': None,
    'tracking.match': None,
    'tracking.robust': None,
    'tracking.track_window': None,
    'tracking.min_new_track_points': None,
    'tracking.min_match_points': None,
    'tracking.img_scale': None,
    'tracking.of_window_size': None,
    'tracking.of_max_levels': None,
    'tracking.save_shifted_instances': None,
    'tracking.kf_node_indices': None,
    'tracking.kf_init_frame_count': None
}

INFO:sleap.nn.inference:Auto-selected GPU 0 with 11662 MiB of free memory.
2024-08-31 11:07:05.171639: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
Versions:
SLEAP: 1.3.4
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Windows-10-10.0.22621-SP0

System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True

Finished inference at: 2024-08-31 11:07:16.914447
Total runtime: 20.598233461380005 secs
Predicted frames: 39/39
Provenance:
{
    'model_paths': [
        'C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\\models\\240831_110334.multi_instance.n=101\\training_config.json'
    ],
    'predictor': 'BottomUpPredictor',
    'sleap_version': '1.3.4',
    'platform': 'Windows-10-10.0.22621-SP0',
    'command': 'C:\\Users\\Liezl\\miniconda3\\envs\\sleap_1.3.4\\Scripts\\sleap-track C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/courtship_labels.slp --only-suggested-frames -m C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\\models\\240831_110334.multi_instance.n=101 -o C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\\predictions\\courtship_labels.slp.240831_110650.predictions.slp --verbosity json --no-empty-frames',
    'data_path': 'C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship/courtship_labels.slp',
    'output_path': 'C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\\predictions\\courtship_labels.slp.240831_110650.predictions.slp',
    'total_elapsed': 20.598233461380005,
    'start_timestamp': '2024-08-31 11:06:56.316214',
    'finish_timestamp': '2024-08-31 11:07:16.914447'
}

Saved output: C:/Users/Liezl/Projects/sleap-estimates-animal-poses/datasets/drosophila-melanogaster-courtship\predictions\courtship_labels.slp.240831_110650.predictions.slp

Process return code: 0




 Terminal: Opening Training GUI

unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'






 sleap-label
![image](https://github.com/user-attachments/assets/f31a7665-250c-48fd-bd4c-7ac30c2b2b15)

(sleap_1.3.4) λ sleap-label
Saving config: C:\Users\Liezl/.sleap/1.3.4/preferences.yaml

Software versions:
SLEAP: 1.3.4
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Windows-10-10.0.22621-SP0

Happy SLEAPing! :)




 mamba create sleap

(base) λ mamba create -y -n sleap_1.3.4 -c conda-forge -c nvidia -c sleap/label/dev -c sleap -c anaconda sleap=1.3.4

                  __    __    __    __
                 /  \  /  \  /  \  /  \
                /    \/    \/    \/    \
███████████████/  /██/  /██/  /██/  /████████████████████████
              /  / \   / \   / \   / \  \____
             /  /   \_/   \_/   \_/   \    o \__,
            / _/                       \_____/  `
            |/
        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝

        mamba (1.4.2) supported by @QuantStack

        GitHub:  https://github.com/mamba-org/mamba
        Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████


Looking for: ['sleap=1.3.4']

sleap/label/dev/win-64                               1.5kB @   4.1kB/s  0.4s
sleap/label/dev/noarch                             135.0 B @ 243.0 B/s  0.2s
nvidia/noarch                                       18.3kB @  31.2kB/s  0.6s
sleap/noarch                                                  No change
anaconda/noarch                                               No change
nvidia/win-64                                      110.3kB @ 111.9kB/s  1.0s
sleap/win-64                                                  No change
anaconda/win-64                                      3.2MB @   2.0MB/s  1.2s
conda-forge/noarch                                  18.8MB @   6.6MB/s  3.5s
conda-forge/win-64                                  28.2MB @   6.2MB/s  5.6s
Transaction

  Prefix: C:\Users\Liezl\miniconda3\envs\sleap_1.3.4

  Updating specs:

   - sleap=1.3.4


  Package                               Version  Build                   Channel                      Size
------------------------------------------------------------------------------------------------------------
  Install:
------------------------------------------------------------------------------------------------------------

  + attrs                                21.4.0  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + brotli                                1.1.0  hcfcfb64_1              conda-forge/win-64         Cached
  + brotli-bin                            1.1.0  hcfcfb64_1              conda-forge/win-64         Cached
  + ca-certificates                   2024.8.30  h56e8100_0              conda-forge/win-64          159kB
  + cached-property                       1.5.2  hd8ed1ab_1              conda-forge/noarch         Cached
  + cached_property                       1.5.2  pyha770c72_1            conda-forge/noarch         Cached
  + cattrs                                1.1.1  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + certifi                            2024.7.4  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + cloudpickle                           2.2.1  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + cuda-nvcc                           11.3.58  hb8d16a4_0              nvidia/win-64              Cached
  + cudatoolkit                          11.3.1  hf2f0253_13             conda-forge/win-64         Cached
  + cudnn                              8.2.1.32  h754d62a_0              conda-forge/win-64         Cached
  + cycler                               0.11.0  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + cytoolz                              0.12.0  py37hcc03f2d_0          conda-forge/win-64         Cached
  + dask-core                          2022.2.0  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + fonttools                            4.38.0  py37h51bd9d9_0          conda-forge/win-64         Cached
  + freeglut                              3.2.2  he0c23c2_3              conda-forge/win-64         Cached
  + freetype                             2.12.1  hdaf720e_2              conda-forge/win-64         Cached
  + fsspec                             2023.1.0  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + geos                                 3.11.0  h39d44d4_0              conda-forge/win-64         Cached
  + h5py                                  3.7.0  nompi_py37h24adfc3_101  conda-forge/win-64         Cached
  + hdf5                                 1.12.2  nompi_h2a0e4a3_101      conda-forge/win-64           11MB
  + icu                                    69.1  h0e60522_0              conda-forge/win-64         Cached
  + imagecodecs-lite                  2019.12.3  py37h0b711f8_5          conda-forge/win-64         Cached
  + imageio                              2.35.1  pyh12aca89_0            conda-forge/noarch         Cached
  + imgaug                                0.4.0  pyhd8ed1ab_1            conda-forge/noarch         Cached
  + importlib-metadata                   4.11.4  py37h03978a9_0          conda-forge/win-64         Cached
  + intel-openmp                       2024.2.1  h57928b3_1083           conda-forge/win-64         Cached
  + jasper                               2.0.33  hc2e4405_1              conda-forge/win-64         Cached
  + joblib                                1.3.2  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + jpeg                                     9e  hcfcfb64_3              conda-forge/win-64         Cached
  + jsmin                                 3.0.1  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + jsonpickle                              1.2  py_0                    conda-forge/noarch         Cached
  + kiwisolver                            1.4.4  py37h8c56517_0          conda-forge/win-64         Cached
  + krb5                                 1.20.1  h6609f42_0              conda-forge/win-64         Cached
  + lcms2                                  2.14  h90d422f_0              conda-forge/win-64         Cached
  + lerc                                  4.0.0  h63175ca_0              conda-forge/win-64         Cached
  + libaec                                1.1.3  h63175ca_0              conda-forge/win-64         Cached
  + libblas                               3.9.0  23_win64_mkl            conda-forge/win-64         Cached
  + libbrotlicommon                       1.1.0  hcfcfb64_1              conda-forge/win-64         Cached
  + libbrotlidec                          1.1.0  hcfcfb64_1              conda-forge/win-64         Cached
  + libbrotlienc                          1.1.0  hcfcfb64_1              conda-forge/win-64         Cached
  + libcblas                              3.9.0  23_win64_mkl            conda-forge/win-64         Cached
  + libclang                             13.0.1  default_h66ee7f4_6      conda-forge/win-64           20MB
  + libcurl                               8.1.2  h68f0423_0              conda-forge/win-64         Cached
  + libdeflate                             1.14  hcfcfb64_0              conda-forge/win-64         Cached
  + libhwloc                             2.11.1  default_h8125262_1000   conda-forge/win-64         Cached
  + libiconv                               1.17  hcfcfb64_2              conda-forge/win-64         Cached
  + liblapack                             3.9.0  23_win64_mkl            conda-forge/win-64         Cached
  + liblapacke                            3.9.0  23_win64_mkl            conda-forge/win-64         Cached
  + libopencv                             4.5.1  py37ha0199f4_0          conda-forge/win-64         Cached
  + libpng                               1.6.43  h19919ed_0              conda-forge/win-64         Cached
  + libprotobuf                          3.21.8  h12be248_0              conda-forge/win-64         Cached
  + libsodium                            1.0.18  h8d14728_1              conda-forge/win-64         Cached
  + libsqlite                            3.46.0  h2466b09_0              conda-forge/win-64         Cached
  + libssh2                              1.10.0  h680486a_3              conda-forge/win-64         Cached
  + libtiff                               4.4.0  hc4f729c_5              conda-forge/win-64         Cached
  + libwebp-base                          1.4.0  hcfcfb64_0              conda-forge/win-64         Cached
  + libxcb                                 1.13  hcd874cb_1004           conda-forge/win-64         Cached
  + libxml2                              2.12.7  h283a6d9_1              conda-forge/win-64         Cached
  + libxslt                              1.1.39  h3df6e99_0              conda-forge/win-64         Cached
  + libzlib                              1.2.13  h2466b09_6              conda-forge/win-64         Cached
  + locket                                1.0.0  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + m2w64-gcc-libgfortran                 5.3.0  6                       conda-forge/win-64         Cached
  + m2w64-gcc-libs                        5.3.0  7                       conda-forge/win-64         Cached
  + m2w64-gcc-libs-core                   5.3.0  7                       conda-forge/win-64         Cached
  + m2w64-gmp                             6.1.0  2                       conda-forge/win-64         Cached
  + m2w64-libwinpthread-git  5.0.0.4634.697f757  2                       conda-forge/win-64         Cached
  + markdown-it-py                        2.2.0  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + matplotlib-base                       3.5.3  py37hbaab90a_2          conda-forge/win-64         Cached
  + mdurl                                 0.1.2  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + mkl                                2024.1.0  h66d3029_694            conda-forge/win-64         Cached
  + msys2-conda-epoch                  20160418  1                       conda-forge/win-64         Cached
  + munkres                               1.1.4  pyh9f0ad1d_0            conda-forge/noarch         Cached
  + networkx                              2.6.3  pyhd8ed1ab_1            conda-forge/noarch         Cached
  + numpy                                1.21.6  py37h2830a78_0          conda-forge/win-64         Cached
  + opencv                                4.5.1  py37h03978a9_0          conda-forge/win-64           22kB
  + openjpeg                              2.5.0  hc9384bd_1              conda-forge/win-64         Cached
  + openssl                              1.1.1w  hcfcfb64_0              conda-forge/win-64         Cached
  + packaging                              23.2  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + pandas                                1.3.5  py37h9386db6_0          conda-forge/win-64         Cached
  + partd                                 1.4.1  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + patsy                                 0.5.6  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + pillow                                9.2.0  py37h42a8222_2          conda-forge/win-64         Cached
  + pip                                    24.0  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + protobuf                             4.21.8  py37h7f67f24_0          conda-forge/win-64         Cached
  + psutil                                5.9.3  py37h51bd9d9_0          conda-forge/win-64         Cached
  + pthread-stubs                           0.4  hcd874cb_1001           conda-forge/win-64         Cached
  + pthreads-win32                        2.9.1  hfa6e2cd_3              conda-forge/win-64         Cached
  + py-opencv                             4.5.1  py37heaed05f_0          conda-forge/win-64           24kB
  + pygments                             2.17.2  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + pykalman                              0.9.7  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + pyparsing                             3.1.4  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + pyside2                              5.13.2  py37h760f651_8          conda-forge/win-64         Cached
  + python                               3.7.12  h7840368_100_cpython    conda-forge/win-64         Cached
  + python-dateutil                       2.9.0  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + python-rapidjson                        1.9  py37h7f67f24_0          conda-forge/win-64         Cached
  + python_abi                              3.7  4_cp37m                 conda-forge/win-64         Cached
  + pytz                                 2024.1  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + pywavelets                            1.3.0  py37h3a130e4_1          conda-forge/win-64         Cached
  + pyyaml                                  6.0  py37hcc03f2d_4          conda-forge/win-64         Cached
  + pyzmq                                24.0.1  py37h7347f05_0          conda-forge/win-64         Cached
  + qt                                   5.12.9  h556501e_6              conda-forge/win-64         Cached
  + qtpy                                  2.4.1  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + rich                                 13.7.1  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + scikit-image                         0.19.3  py37h3182a2c_1          conda-forge/win-64         Cached
  + scikit-learn                            1.0  py37ha78be43_1          conda-forge/win-64         Cached
  + scikit-video                         1.1.11  pyh24bf2e0_0            conda-forge/noarch         Cached
  + scipy                                 1.7.3  py37hb6553fb_0          conda-forge/win-64         Cached
  + seaborn                              0.12.2  hd8ed1ab_0              conda-forge/noarch         Cached
  + seaborn-base                         0.12.2  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + setuptools                           59.8.0  py37h03978a9_1          conda-forge/win-64         Cached
  + shapely                               1.8.5  py37h475e9a0_0          conda-forge/win-64         Cached
  + six                                  1.16.0  pyh6c4a22f_0            conda-forge/noarch         Cached
  + sleap                                 1.3.4  py37_2                  sleap/label/dev/win-64        7MB
  + sqlite                               3.46.0  h2466b09_0              conda-forge/win-64         Cached
  + statsmodels                          0.13.2  py37h3a130e4_0          conda-forge/win-64         Cached
  + tbb                               2021.12.0  hc790b64_4              conda-forge/win-64         Cached
  + tensorflow                            2.7.0  py37h5685391_4          sleap/label/dev/win-64     Cached
  + tensorflow-hub                       0.12.0  pyhca92ed8_0            conda-forge/noarch         Cached
  + threadpoolctl                         3.1.0  pyh8a188c0_0            conda-forge/noarch         Cached
  + tifffile                           2020.6.3  py_0                    conda-forge/noarch         Cached
  + tk                                   8.6.13  h5226925_1              conda-forge/win-64         Cached
  + toolz                                0.12.1  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + typing-extensions                     4.7.1  hd8ed1ab_0              conda-forge/noarch         Cached
  + typing_extensions                     4.7.1  pyha770c72_0            conda-forge/noarch         Cached
  + ucrt                           10.0.22621.0  h57928b3_0              conda-forge/win-64         Cached
  + unicodedata2                         14.0.0  py37hcc03f2d_1          conda-forge/win-64         Cached
  + vc                                     14.3  h8a93ad2_20             conda-forge/win-64         Cached
  + vc14_runtime                    14.40.33810  hcc2c482_20             conda-forge/win-64         Cached
  + vs2015_runtime                  14.40.33810  h3bf8584_20             conda-forge/win-64         Cached
  + wheel                                0.42.0  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + xorg-libxau                          1.0.11  hcd874cb_0              conda-forge/win-64         Cached
  + xorg-libxdmcp                         1.1.3  hcd874cb_0              conda-forge/win-64         Cached
  + xz                                    5.2.6  h8d14728_0              conda-forge/win-64         Cached
  + yaml                                  0.2.5  h8ffe710_2              conda-forge/win-64         Cached
  + zeromq                                4.3.4  h0e60522_1              conda-forge/win-64         Cached
  + zipp                                 3.15.0  pyhd8ed1ab_0            conda-forge/noarch         Cached
  + zlib                                 1.2.13  h2466b09_6              conda-forge/win-64         Cached
  + zstd                                  1.5.6  h0ea2cb4_0              conda-forge/win-64         Cached

  Summary:

  Install: 140 packages

  Total download: 39MB




Mac


 GUI Training

image

image


 Terminal: Opening Training GUI

2024-08-31 10:38:42.835 python[28887:4467799] +[CATransaction synchronize] called within transaction
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
unhashable type: 'dict'
qt.qpa.fonts: Populating font family aliases took 123 ms. Replace uses of missing font family ".AppleSystemUIFont" with one that exists to avoid this cost.




 Terminal: Training

Resetting monitor window.
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103/viz/validation.*.png
Start training centroid...
['sleap-train', '/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpdf8ag4oh/240831_104002_training_job.json', '/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp', '--zmq', '--save_viz']
INFO:sleap.nn.training:Versions:
SLEAP: 1.3.4
TensorFlow: 2.9.2
Numpy: 1.22.4
Python: 3.9.15
OS: macOS-13.5-arm64-arm-64bit
INFO:sleap.nn.training:Training labels file: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Training profile: /var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpdf8ag4oh/240831_104002_training_job.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
    "training_job_path": "/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpdf8ag4oh/240831_104002_training_job.json",
    "labels_path": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp",
    "video_paths": [
        ""
    ],
    "val_labels": null,
    "test_labels": null,
    "base_checkpoint": null,
    "tensorboard": false,
    "save_viz": true,
    "zmq": true,
    "run_name": "",
    "prefix": "",
    "suffix": "",
    "cpu": false,
    "first_gpu": false,
    "last_gpu": false,
    "gpu": "auto"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
    "data": {
        "labels": {
            "training_labels": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp",
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": [
                12,
                98,
                101,
                71,
                2,
                95,
                86,
                50,
                63,
                90,
                87,
                37,
                58,
                13,
                43,
                16,
                40,
                24,
                82,
                44,
                67,
                10,
                26,
                84,
                49,
                6,
                48,
                53,
                100,
                47,
                61,
                21,
                15,
                52,
                75,
                30,
                59,
                72,
                56,
                80,
                5,
                102,
                94,
                27,
                69,
                74,
                38,
                73,
                29,
                20,
                33,
                42,
                81,
                14,
                62,
                78,
                70,
                7,
                64,
                45,
                77,
                85,
                3,
                54,
                51,
                99,
                39,
                31,
                46,
                66,
                93,
                19,
                83,
                11,
                0,
                92,
                25,
                8,
                60,
                22,
                28,
                36,
                1,
                9,
                55,
                57,
                17,
                88,
                65,
                97,
                96,
                23,
                41
            ],
            "validation_inds": [
                76,
                79,
                4,
                89,
                34,
                35,
                32,
                91,
                68,
                18
            ],
            "test_inds": null,
            "search_path_hints": [
                "",
                "",
                "",
                "",
                ""
            ],
            "skeletons": []
        },
        "preprocessing": {
            "ensure_rgb": false,
            "ensure_grayscale": false,
            "imagenet_mode": null,
            "input_scaling": 0.5,
            "pad_to_stride": 16,
            "resize_and_pad_to_target": true,
            "target_height": 1024,
            "target_width": 1024
        },
        "instance_cropping": {
            "center_on_part": "thorax",
            "crop_size": null,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 16,
                "output_stride": 2,
                "filters": 16,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": null,
            "centroid": {
                "anchor_part": "thorax",
                "sigma": 2.5,
                "output_stride": 2,
                "loss_weight": 1.0,
                "offset_refinement": false
            },
            "centered_instance": null,
            "multi_instance": null,
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        },
        "base_checkpoint": null
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -180.0,
            "rotation_max_angle": 180.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": false,
            "flip_horizontal": false
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": 200,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": 10,
        "min_val_batches_per_epoch": 10,
        "epochs": 2,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 20
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "240831_104002.centroid.n=103",
        "run_name_prefix": "",
        "run_name_suffix": "",
        "runs_folder": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": true,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": true,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.4.1a2",
    "filename": "/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpdf8ag4oh/240831_104002_training_job.json"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU.
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training:  Splits: Training = 93 / Validation = 10.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
Metal device set to: Apple M2 Pro

systemMemory: 16.00 GB
maxCacheSize: 5.33 GB

2024-08-31 10:40:07.306830: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2024-08-31 10:40:07.307008: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
2024-08-31 10:40:07.579919: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
INFO:sleap.nn.training:Loaded test example. [0.908s]
INFO:sleap.nn.training:  Input shape: (512, 512, 1)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training:  Backbone: UNet(stacks=1, filters=16, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=3, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training:  Max stride: 16
INFO:sleap.nn.training:  Parameters: 1,953,105
INFO:sleap.nn.training:  Heads: 
INFO:sleap.nn.training:    [0] = CentroidConfmapsHead(anchor_part='thorax', sigma=2.5, output_stride=2, loss_weight=1.0)
INFO:sleap.nn.training:  Outputs: 
INFO:sleap.nn.training:    [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 256, 256, 1), dtype=tf.float32, name=None), name='CentroidConfmapsHead/BiasAdd:0', description="created by layer 'CentroidConfmapsHead'")
INFO:sleap.nn.training:Training from scratch
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 93
INFO:sleap.nn.training:Validation set: n = 10
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training:  Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training:  Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=20)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.callbacks:Training controller subscribed to: tcp://127.0.0.1:9000 (topic: )
INFO:sleap.nn.training:  ZMQ controller subcribed to: tcp://127.0.0.1:9000
INFO:sleap.nn.callbacks:Progress reporter publishing on: tcp://127.0.0.1:9001 for: not_set
INFO:sleap.nn.training:  ZMQ progress reporter publish on: tcp://127.0.0.1:9001
INFO:sleap.nn.training:Created run path: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103
INFO:sleap.nn.training:Setting up visualization...
2024-08-31 10:40:09.211851: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:40:09.239146: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:40:09.252665: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -34 } dim { size: -35 } dim { size: -36 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -37 } dim { size: -38 } dim { size: 1 } } }
2024-08-31 10:40:09.730678: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:40:09.758446: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:40:09.771833: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -34 } dim { size: -35 } dim { size: -36 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -37 } dim { size: -38 } dim { size: 1 } } }
INFO:sleap.nn.training:Finished trainer set up. [2.6s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
INFO:sleap.nn.training:Finished creating training datasets. [4.0s]
INFO:sleap.nn.training:Starting training loop...
Epoch 1/2
2024-08-31 10:40:14.150629: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:40:50.149551: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
200/200 - 38s - loss: 2.7716e-04 - val_loss: 1.5770e-04 - lr: 1.0000e-04 - 38s/epoch - 191ms/step
Epoch 2/2
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103/viz/validation.*.png
200/200 - 36s - loss: 9.0670e-05 - val_loss: 4.4644e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step
INFO:sleap.nn.training:Finished training loop. [1.2 min]
INFO:sleap.nn.training:Deleting visualization directory: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103/viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103/viz/validation.*.png
2024-08-31 10:41:29.785486: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:41:29.898676: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -43 } dim { size: -44 } dim { size: -45 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -9 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: -46 } dim { size: -47 } dim { size: 1 } } }
2024-08-31 10:41:29.898854: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 4 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -9 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: -55 } dim { size: -56 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸  99% ETA: 0:00:01 35.1 FPS2024-08-31 10:41:33.282831: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:41:33.395526: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -56 } dim { size: -57 } dim { size: -58 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -9 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: -59 } dim { size: -60 } dim { size: 1 } } }
2024-08-31 10:41:33.395803: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: -18 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -9 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: -68 } dim { size: -69 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 25.5 FPS
INFO:sleap.nn.evals:Saved predictions: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103/labels_pr.train.slp
INFO:sleap.nn.evals:Saved metrics: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103/metrics.train.npz
INFO:sleap.nn.evals:OKS mAP: 0.980089
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2024-08-31 10:41:34.744262: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:41:34.856012: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -43 } dim { size: -44 } dim { size: -45 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -9 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: -46 } dim { size: -47 } dim { size: 1 } } }
2024-08-31 10:41:34.856193: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 4 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -9 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: -55 } dim { size: -56 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━  80% ETA: 0:00:01 47.6 FPS2024-08-31 10:41:36.014392: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:41:36.124409: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -56 } dim { size: -57 } dim { size: -58 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -9 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: -59 } dim { size: -60 } dim { size: 1 } } }
2024-08-31 10:41:36.124641: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: -18 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -9 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -9 } dim { size: -68 } dim { size: -69 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 4.8 FPS
INFO:sleap.nn.evals:Saved predictions: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103/labels_pr.val.slp
INFO:sleap.nn.evals:Saved metrics: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103/metrics.val.npz
INFO:sleap.nn.evals:OKS mAP: 1.000000
INFO:sleap.nn.callbacks:Closing the reporter controller/context.
INFO:sleap.nn.callbacks:Closing the training controller socket/context.
Run Path: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103
Finished training centroid.
Resetting monitor window.
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104137.centered_instance.n=103/viz/validation.*.png
Start training centered_instance...
['sleap-train', '/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpgxv9utkl/240831_104137_training_job.json', '/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp', '--zmq', '--save_viz']
INFO:sleap.nn.training:Versions:
SLEAP: 1.3.4
TensorFlow: 2.9.2
Numpy: 1.22.4
Python: 3.9.15
OS: macOS-13.5-arm64-arm-64bit
INFO:sleap.nn.training:Training labels file: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Training profile: /var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpgxv9utkl/240831_104137_training_job.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
    "training_job_path": "/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpgxv9utkl/240831_104137_training_job.json",
    "labels_path": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp",
    "video_paths": [
        ""
    ],
    "val_labels": null,
    "test_labels": null,
    "base_checkpoint": null,
    "tensorboard": false,
    "save_viz": true,
    "zmq": true,
    "run_name": "",
    "prefix": "",
    "suffix": "",
    "cpu": false,
    "first_gpu": false,
    "last_gpu": false,
    "gpu": "auto"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
    "data": {
        "labels": {
            "training_labels": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp",
            "validation_labels": null,
            "validation_fraction": 0.1,
            "test_labels": null,
            "split_by_inds": false,
            "training_inds": [
                6,
                82,
                95,
                24,
                68,
                92,
                64,
                30,
                7,
                87,
                19,
                100,
                73,
                23,
                96,
                13,
                42,
                58,
                86,
                54,
                28,
                79,
                34,
                81,
                93,
                27,
                49,
                1,
                51,
                74,
                91,
                60,
                31,
                46,
                77,
                75,
                98,
                78,
                10,
                63,
                45,
                32,
                29,
                16,
                84,
                71,
                36,
                48,
                39,
                0,
                88,
                4,
                25,
                12,
                44,
                50,
                5,
                69,
                97,
                101,
                102,
                33,
                80,
                89,
                70,
                55,
                85,
                38,
                76,
                59,
                62,
                40,
                99,
                11,
                94,
                90,
                15,
                53,
                2,
                21,
                18,
                72,
                83,
                3,
                57,
                41,
                66,
                52,
                9,
                20,
                35,
                65,
                14
            ],
            "validation_inds": [
                61,
                43,
                26,
                67,
                47,
                56,
                17,
                22,
                8,
                37
            ],
            "test_inds": null,
            "search_path_hints": [
                "",
                "",
                "",
                ""
            ],
            "skeletons": []
        },
        "preprocessing": {
            "ensure_rgb": false,
            "ensure_grayscale": false,
            "imagenet_mode": null,
            "input_scaling": 1.0,
            "pad_to_stride": 1,
            "resize_and_pad_to_target": true,
            "target_height": 1024,
            "target_width": 1024
        },
        "instance_cropping": {
            "center_on_part": "thorax",
            "crop_size": 144,
            "crop_size_detection_padding": 16
        }
    },
    "model": {
        "backbone": {
            "leap": null,
            "unet": {
                "stem_stride": null,
                "max_stride": 16,
                "output_stride": 4,
                "filters": 24,
                "filters_rate": 2.0,
                "middle_block": true,
                "up_interpolate": true,
                "stacks": 1
            },
            "hourglass": null,
            "resnet": null,
            "pretrained_encoder": null
        },
        "heads": {
            "single_instance": null,
            "centroid": null,
            "centered_instance": {
                "anchor_part": "thorax",
                "part_names": [
                    "head",
                    "thorax",
                    "abdomen",
                    "wingL",
                    "wingR",
                    "forelegL4",
                    "forelegR4",
                    "midlegL4",
                    "midlegR4",
                    "hindlegL4",
                    "hindlegR4",
                    "eyeL",
                    "eyeR"
                ],
                "sigma": 2.5,
                "output_stride": 4,
                "loss_weight": 1.0,
                "offset_refinement": false
            },
            "multi_instance": null,
            "multi_class_bottomup": null,
            "multi_class_topdown": null
        },
        "base_checkpoint": null
    },
    "optimization": {
        "preload_data": true,
        "augmentation_config": {
            "rotate": true,
            "rotation_min_angle": -180.0,
            "rotation_max_angle": 180.0,
            "translate": false,
            "translate_min": -5,
            "translate_max": 5,
            "scale": false,
            "scale_min": 0.9,
            "scale_max": 1.1,
            "uniform_noise": false,
            "uniform_noise_min_val": 0.0,
            "uniform_noise_max_val": 10.0,
            "gaussian_noise": false,
            "gaussian_noise_mean": 5.0,
            "gaussian_noise_stddev": 1.0,
            "contrast": false,
            "contrast_min_gamma": 0.5,
            "contrast_max_gamma": 2.0,
            "brightness": false,
            "brightness_min_val": 0.0,
            "brightness_max_val": 10.0,
            "random_crop": false,
            "random_crop_height": 256,
            "random_crop_width": 256,
            "random_flip": false,
            "flip_horizontal": false
        },
        "online_shuffling": true,
        "shuffle_buffer_size": 128,
        "prefetch": true,
        "batch_size": 4,
        "batches_per_epoch": 200,
        "min_batches_per_epoch": 200,
        "val_batches_per_epoch": 10,
        "min_val_batches_per_epoch": 10,
        "epochs": 2,
        "optimizer": "adam",
        "initial_learning_rate": 0.0001,
        "learning_rate_schedule": {
            "reduce_on_plateau": true,
            "reduction_factor": 0.5,
            "plateau_min_delta": 1e-06,
            "plateau_patience": 5,
            "plateau_cooldown": 3,
            "min_learning_rate": 1e-08
        },
        "hard_keypoint_mining": {
            "online_mining": false,
            "hard_to_easy_ratio": 2.0,
            "min_hard_keypoints": 2,
            "max_hard_keypoints": null,
            "loss_scale": 5.0
        },
        "early_stopping": {
            "stop_training_on_plateau": true,
            "plateau_min_delta": 1e-08,
            "plateau_patience": 10
        }
    },
    "outputs": {
        "save_outputs": true,
        "run_name": "240831_104137.centered_instance.n=103",
        "run_name_prefix": "",
        "run_name_suffix": "",
        "runs_folder": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models",
        "tags": [
            ""
        ],
        "save_visualizations": true,
        "delete_viz_images": true,
        "zip_outputs": false,
        "log_to_csv": true,
        "checkpointing": {
            "initial_model": false,
            "best_model": true,
            "every_epoch": false,
            "latest_model": false,
            "final_model": false
        },
        "tensorboard": {
            "write_logs": false,
            "loss_frequency": "epoch",
            "architecture_graph": false,
            "profile_graph": false,
            "visualizations": true
        },
        "zmq": {
            "subscribe_to_controller": true,
            "controller_address": "tcp://127.0.0.1:9000",
            "controller_polling_timeout": 10,
            "publish_updates": true,
            "publish_address": "tcp://127.0.0.1:9001"
        }
    },
    "name": "",
    "description": "",
    "sleap_version": "1.4.1a2",
    "filename": "/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpgxv9utkl/240831_104137_training_job.json"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU.
INFO:sleap.nn.training:Using GPU 0 for acceleration.
INFO:sleap.nn.training:Disabled GPU memory pre-allocation.
INFO:sleap.nn.training:System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training:  Splits: Training = 93 / Validation = 10.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
Metal device set to: Apple M2 Pro

systemMemory: 16.00 GB
maxCacheSize: 5.33 GB

2024-08-31 10:41:42.512151: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2024-08-31 10:41:42.512299: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
2024-08-31 10:41:42.781321: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
2024-08-31 10:41:43.568742: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 1 } } }
INFO:sleap.nn.training:Loaded test example. [1.192s]
INFO:sleap.nn.training:  Input shape: (144, 144, 1)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training:  Backbone: UNet(stacks=1, filters=24, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=2, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training:  Max stride: 16
INFO:sleap.nn.training:  Parameters: 4,311,445
INFO:sleap.nn.training:  Heads: 
INFO:sleap.nn.training:    [0] = CenteredInstanceConfmapsHead(part_names=['head', 'thorax', 'abdomen', 'wingL', 'wingR', 'forelegL4', 'forelegR4', 'midlegL4', 'midlegR4', 'hindlegL4', 'hindlegR4', 'eyeL', 'eyeR'], anchor_part='thorax', sigma=2.5, output_stride=4, loss_weight=1.0)
INFO:sleap.nn.training:  Outputs: 
INFO:sleap.nn.training:    [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 36, 36, 13), dtype=tf.float32, name=None), name='CenteredInstanceConfmapsHead/BiasAdd:0', description="created by layer 'CenteredInstanceConfmapsHead'")
INFO:sleap.nn.training:Training from scratch
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 93
INFO:sleap.nn.training:Validation set: n = 10
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training:  Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training:  Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=10)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.callbacks:Training controller subscribed to: tcp://127.0.0.1:9000 (topic: )
INFO:sleap.nn.training:  ZMQ controller subcribed to: tcp://127.0.0.1:9000
INFO:sleap.nn.callbacks:Progress reporter publishing on: tcp://127.0.0.1:9001 for: not_set
INFO:sleap.nn.training:  ZMQ progress reporter publish on: tcp://127.0.0.1:9001
INFO:sleap.nn.training:Created run path: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104137.centered_instance.n=103
INFO:sleap.nn.training:Setting up visualization...
2024-08-31 10:41:44.317850: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 1 } } }
2024-08-31 10:41:44.811632: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 1 } } }
INFO:sleap.nn.training:Finished trainer set up. [2.4s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
2024-08-31 10:41:48.008680: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 1 } } }
2024-08-31 10:41:49.108974: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 1 } } }
INFO:sleap.nn.training:Finished creating training datasets. [4.4s]
INFO:sleap.nn.training:Starting training loop...
2024-08-31 10:41:49.349177: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 1 } } }
Epoch 1/2
2024-08-31 10:41:49.820443: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:42:01.214235: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 1 } } }
2024-08-31 10:42:01.392528: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:42:02.320142: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:42:02.327786: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 13 } dim { size: 36 } dim { size: 36 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -5 } dim { size: -6 } dim { size: 1 } } }
200/200 - 14s - loss: 0.0111 - head: 0.0076 - thorax: 0.0065 - abdomen: 0.0115 - wingL: 0.0127 - wingR: 0.0129 - forelegL4: 0.0112 - forelegR4: 0.0110 - midlegL4: 0.0133 - midlegR4: 0.0133 - hindlegL4: 0.0132 - hindlegR4: 0.0132 - eyeL: 0.0090 - eyeR: 0.0092 - val_loss: 0.0090 - val_head: 0.0035 - val_thorax: 0.0045 - val_abdomen: 0.0082 - val_wingL: 0.0098 - val_wingR: 0.0100 - val_forelegL4: 0.0106 - val_forelegR4: 0.0088 - val_midlegL4: 0.0122 - val_midlegR4: 0.0123 - val_hindlegL4: 0.0126 - val_hindlegR4: 0.0128 - val_eyeL: 0.0055 - val_eyeR: 0.0059 - lr: 1.0000e-04 - 14s/epoch - 68ms/step
Epoch 2/2
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104137.centered_instance.n=103/viz/validation.*.png
2024-08-31 10:42:13.946977: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 1 } } }
200/200 - 12s - loss: 0.0074 - head: 0.0025 - thorax: 0.0032 - abdomen: 0.0063 - wingL: 0.0072 - wingR: 0.0082 - forelegL4: 0.0086 - forelegR4: 0.0085 - midlegL4: 0.0110 - midlegR4: 0.0109 - hindlegL4: 0.0116 - hindlegR4: 0.0115 - eyeL: 0.0034 - eyeR: 0.0035 - val_loss: 0.0061 - val_head: 0.0017 - val_thorax: 0.0021 - val_abdomen: 0.0054 - val_wingL: 0.0060 - val_wingR: 0.0069 - val_forelegL4: 0.0074 - val_forelegR4: 0.0066 - val_midlegL4: 0.0092 - val_midlegR4: 0.0094 - val_hindlegL4: 0.0098 - val_hindlegR4: 0.0101 - val_eyeL: 0.0024 - val_eyeR: 0.0022 - lr: 1.0000e-04 - 12s/epoch - 58ms/step
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103/viz/validation.*.png
Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104137.centered_instance.n=103/viz/validation.*.png
INFO:sleap.nn.training:Finished training loop. [0.4 min]
INFO:sleap.nn.training:Deleting visualization directory: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104137.centered_instance.n=103/viz
INFO:sleap.nn.training:Saving evaluation metrics to model folder...
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2024-08-31 10:42:15.583669: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:42:15.644206: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 4 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -25 } dim { size: -26 } dim { size: 1 } } }
2024-08-31 10:42:15.649281: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -42 } dim { size: -43 } dim { size: -44 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -10 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -10 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -10 } dim { size: -46 } dim { size: -47 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸  99% ETA: 0:00:01 39.8 FPS2024-08-31 10:42:18.510483: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:42:18.570343: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: -13 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -39 } dim { size: -40 } dim { size: 1 } } }
2024-08-31 10:42:18.574968: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -57 } dim { size: -58 } dim { size: -59 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -10 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -10 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -10 } dim { size: -61 } dim { size: -62 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 31.9 FPS
INFO:sleap.nn.evals:Saved predictions: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104137.centered_instance.n=103/labels_pr.train.slp
INFO:sleap.nn.evals:Saved metrics: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104137.centered_instance.n=103/metrics.train.npz
INFO:sleap.nn.evals:OKS mAP: 0.003687
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2024-08-31 10:42:19.595013: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:42:19.653563: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 4 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -25 } dim { size: -26 } dim { size: 1 } } }
2024-08-31 10:42:19.658071: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -42 } dim { size: -43 } dim { size: -44 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -10 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -10 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -10 } dim { size: -46 } dim { size: -47 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━  80% ETA: 0:00:01 52.0 FPS2024-08-31 10:42:20.233551: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:42:20.298026: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: -13 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -39 } dim { size: -40 } dim { size: 1 } } }
2024-08-31 10:42:20.303196: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -57 } dim { size: -58 } dim { size: -59 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -10 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -10 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -10 } dim { size: -61 } dim { size: -62 } dim { size: 1 } } }
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 9.0 FPS
INFO:sleap.nn.evals:Saved predictions: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104137.centered_instance.n=103/labels_pr.val.slp
INFO:sleap.nn.evals:Saved metrics: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104137.centered_instance.n=103/metrics.val.npz
INFO:sleap.nn.evals:OKS mAP: 0.025248
INFO:sleap.nn.callbacks:Closing the reporter controller/context.
INFO:sleap.nn.callbacks:Closing the training controller socket/context.
Run Path: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104137.centered_instance.n=103
Finished training centered_instance.
Command line call:
sleap-track /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp --only-suggested-frames -m /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103 -m /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104137.centered_instance.n=103 -o /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/predictions/courtship_labels.slp.240831_104221.predictions.slp --verbosity json --no-empty-frames

Started inference at: 2024-08-31 10:42:25.700490
Args:
{
│   'data_path': '/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.slp',
│   'models': [
│   │   '/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104002.centroid.n=103',
│   │   '/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/240831_104137.centered_instance.n=103'
│   ],
│   'frames': '',
│   'only_labeled_frames': False,
│   'only_suggested_frames': True,
│   'output': '/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/predictions/courtship_labels.slp.240831_104221.predictions.slp',
2024-08-31 10:42:26.318598: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2024-08-31 10:42:26.318737: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
│   'no_empty_frames': True,
│   'verbosity': 'json',
│   'video.dataset': None,
│   'video.input_format': 'channels_last',
│   'video.index': '',
│   'cpu': False,
│   'first_gpu': False,
│   'last_gpu': False,
│   'gpu': 'auto',
│   'max_edge_length_ratio': 0.25,
│   'dist_penalty_weight': 1.0,
│   'batch_size': 4,
│   'open_in_gui': False,
│   'peak_threshold': 0.2,
│   'max_instances': None,
│   'tracking.tracker': None,
│   'tracking.max_tracking': None,
│   'tracking.max_tracks': None,
│   'tracking.target_instance_count': None,
│   'tracking.pre_cull_to_target': None,
2024-08-31 10:42:27.299442: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
│   'tracking.pre_cull_iou_threshold': None,
│   'tracking.post_connect_single_breaks': None,
│   'tracking.clean_instance_count': None,
│   'tracking.clean_iou_threshold': None,
│   'tracking.similarity': None,
│   'tracking.match': None,
│   'tracking.robust': None,
│   'tracking.track_window': None,
│   'tracking.min_new_track_points': None,
│   'tracking.min_match_points': None,
│   'tracking.img_scale': None,
│   'tracking.of_window_size': None,
│   'tracking.of_max_levels': None,
│   'tracking.save_shifted_instances': None,
│   'tracking.kf_node_indices': None,
│   'tracking.kf_init_frame_count': None
}

INFO:sleap.nn.inference:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU.
Metal device set to: Apple M2 Pro
2024-08-31 10:42:29.162865: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2024-08-31 10:42:29.221760: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -45 } dim { size: -46 } dim { size: -47 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -15 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: -48 } dim { size: -49 } dim { size: 1 } } }
2024-08-31 10:42:29.221963: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 4 } dim { size: 1024 } dim { size: 1024 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -15 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: -56 } dim { size: -57 } dim { size: 1 } } }
2024-08-31 10:42:29.224345: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -91 } dim { size: -92 } dim { size: -93 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -20 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -20 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -20 } dim { size: -95 } dim { size: -96 } dim { size: 1 } } }
Versions:
SLEAP: 1.3.4
TensorFlow: 2.9.2
Numpy: 1.22.4
Python: 3.9.15
OS: macOS-13.5-arm64-arm-64bit

System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True

Process return code: 0







 sleap-label

image


(sleap_1.3) liezlmaree:~$sleap-label
Saving config: /Users/liezlmaree/.sleap/1.3.4/preferences.yaml
Restoring GUI state...

Software versions:
SLEAP: 1.3.4
TensorFlow: 2.9.2
Numpy: 1.22.4
Python: 3.9.15
OS: macOS-13.5-arm64-arm-64bit

Happy SLEAPing! :)




mm create sleap

liezlmaree:~$mm create -y -n sleap_1.3.4 -c conda-forge -c nvidia -c sleap/label/dev -c sleap -c anaconda sleap=1.3.4
nvidia/noarch (check zst)                           Checked  0.2s
nvidia/osx-arm64 (check zst)                        Checked  0.3s
sleap/osx-arm64 (check zst)                         Checked  1.0s
sleap/label/dev/osx-arm64 (check zst)               Checked  1.1s
sleap/noarch (check zst)                            Checked  2.0s
sleap/label/dev/noarch (check zst)                  Checked  2.8s
nvidia/osx-arm64                                              No change
nvidia/noarch                                       14.7kB @ 123.3kB/s  0.1s
sleap/osx-arm64                                               No change
sleap/label/dev/noarch                             116.0 B @ 215.0 B/s  0.4s
sleap/label/dev/osx-arm64                          962.0 B @   1.3kB/s  0.7s
anaconda/noarch                                               No change
sleap/noarch                                                  No change
anaconda/osx-arm64                                            No change
conda-forge/osx-arm64                               11.4MB @ 788.8kB/s 14.5s
conda-forge/noarch                                  16.2MB @   1.1MB/s 14.7s

Transaction

  Prefix: /Users/liezlmaree/micromamba/envs/sleap_1.3.4

  Updating specs:

   - sleap=1.3.4


warning  libmamba Invalid package cache, file '/Users/liezlmaree/micromamba/pkgs/pyside2-5.15.8-py39h0adaba8_2/lib/python3.9/site-packages/shiboken2/files.dir/shibokensupport/signature/mapping.py' has incorrect size
warning  libmamba Extracted package cache '/Users/liezlmaree/micromamba/pkgs/sleap-1.3.4-py39_0' has invalid size
warning  libmamba Extracted package cache '/Users/liezlmaree/micromamba/pkgs/sleap-1.3.4-py39_0' has invalid SHA-256 checksum
  Package                           Version  Build                   Channel               Size
─────────────────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────────────────

  + bzip2                             1.0.8  h99b78c6_7              conda-forge         Cached
  + libffi                            3.4.2  h3422bc3_5              conda-forge         Cached
  + libzlib                           1.3.1  hfb2fe0b_1              conda-forge         Cached
  + libcxx                           18.1.8  h3ed4263_6              conda-forge         Cached
  + python_abi                          3.9  5_cp39                  conda-forge         Cached
  + giflib                            5.2.2  h93a5062_0              conda-forge         Cached
  + jpeg                                 9e  h1a8c8d9_3              conda-forge         Cached
  + ncurses                             6.5  h7bae524_1              conda-forge         Cached
  + xz                                5.2.6  h57fd34a_0              conda-forge         Cached
  + ca-certificates               2024.8.30  hf0a4a13_0              conda-forge         Cached
  + libdeflate                         1.14  h1a8c8d9_0              conda-forge         Cached
  + libwebp-base                      1.4.0  h93a5062_0              conda-forge         Cached
  + c-ares                           1.33.1  hd74edd7_0              conda-forge         Cached
  + libiconv                           1.17  h0d3ecfb_2              conda-forge         Cached
  + llvm-openmp                      18.1.8  hde57baf_1              conda-forge         Cached
  + libev                              4.33  h93a5062_2              conda-forge         Cached
  + libexpat                          2.6.2  hebf3989_0              conda-forge         Cached
  + nettle                            3.9.1  h40ed0f5_0              conda-forge         Cached
  + lame                              3.100  h1a8c8d9_1003           conda-forge         Cached
  + x264                         1!164.3095  h57fd34a_2              conda-forge         Cached
  + xorg-libxdmcp                     1.1.3  h27ca646_0              conda-forge         Cached
  + libtasn1                         4.19.0  h1a8c8d9_0              conda-forge         Cached
  + pthread-stubs                       0.4  h27ca646_1001           conda-forge         Cached
  + xorg-libxau                      1.0.11  hb547adb_0              conda-forge         Cached
  + libsodium                        1.0.18  h27ca646_1              conda-forge         Cached
  + yaml                              0.2.5  h3422bc3_2              conda-forge         Cached
  + libogg                            1.3.5  h99b78c6_0              conda-forge         Cached
  + libopus                           1.3.1  h27ca646_1              conda-forge         Cached
  + libbrotlicommon                   1.1.0  hb547adb_1              conda-forge         Cached
  + libunistring                     0.9.10  h3422bc3_0              conda-forge         Cached
  + pcre2                             10.44  h297a79d_2              conda-forge         Cached
  + zstd                              1.5.6  hb46c0d2_0              conda-forge         Cached
  + tk                               8.6.13  h5083fa2_1              conda-forge         Cached
  + libpng                           1.6.43  h091b4b1_0              conda-forge         Cached
  + libsqlite                        3.46.0  hfb93653_0              conda-forge         Cached
  + libasprintf                      0.22.5  h8414b35_3              conda-forge         Cached
  + libllvm14                        14.0.6  hd1a9a77_4              conda-forge         Cached
  + nspr                               4.35  hb7217d7_0              conda-forge         Cached
  + geos                             3.12.2  h00cdb27_1              conda-forge         Cached
  + x265                                3.5  hbc6ce65_3              conda-forge         Cached
  + openh264                          2.3.1  hb7217d7_2              conda-forge         Cached
  + libvpx                           1.11.0  hc470f4d_3              conda-forge         Cached
  + gmp                               6.3.0  h7bae524_2              conda-forge         Cached
  + libaec                            1.1.3  hebf3989_0              conda-forge         Cached
  + pixman                           0.43.4  hebf3989_0              conda-forge         Cached
  + re2                          2022.06.01  h9a09cb3_1              conda-forge         Cached
  + graphite2                        1.3.13  hebf3989_1003           conda-forge         Cached
  + lerc                              4.0.0  h9a09cb3_0              conda-forge         Cached
  + svt-av1                           1.4.1  h7ea286d_0              conda-forge         Cached
  + aom                               3.5.0  h7ea286d_0              conda-forge         Cached
  + snappy                           1.1.10  hd04f947_1              conda-forge         Cached
  + libprotobuf                      3.20.3  hb5ab8b9_0              conda-forge         Cached
  + libabseil                    20211102.0  cxx17_h28b99d4_3        conda-forge         Cached
  + icu                                70.1  h6b3803e_0              conda-forge         Cached
  + jasper                           2.0.33  hc3cd1e9_1              conda-forge         Cached
  + libedit                    3.1.20191231  hc8eb9b7_2              conda-forge         Cached
  + readline                            8.2  h92ec313_1              conda-forge         Cached
  + openssl                          1.1.1w  h53f4e23_0              conda-forge         Cached
  + libintl                          0.22.5  h8414b35_3              conda-forge         Cached
  + libgfortran5                     13.2.0  hf226fd6_3              conda-forge         Cached
  + expat                             2.6.2  hebf3989_0              conda-forge         Cached
  + p11-kit                          0.24.1  h29577a5_0              conda-forge         Cached
  + libxcb                             1.13  h9b22ae9_1004           conda-forge         Cached
  + zeromq                            4.3.5  hebf3989_1              conda-forge         Cached
  + libvorbis                         1.3.7  h9f76cd9_0              conda-forge         Cached
  + libbrotlienc                      1.1.0  hb547adb_1              conda-forge         Cached
  + libbrotlidec                      1.1.0  hb547adb_1              conda-forge         Cached
  + freetype                         2.12.1  hadb7bae_2              conda-forge         Cached
  + libasprintf-devel                0.22.5  h8414b35_3              conda-forge         Cached
  + libclang13                       14.0.6  default_hc7183e1_1      conda-forge         Cached
  + nss                               3.104  hd1ce637_0              conda-forge            2MB
  + libtiff                           4.4.0  heb92581_5              conda-forge         Cached
  + abseil-cpp                   20211102.0  he4e09e4_3              conda-forge         Cached
  + libxml2                          2.10.3  h67585b2_4              conda-forge         Cached
  + sqlite                           3.46.0  h5838104_0              conda-forge         Cached
  + mysql-common                     8.0.32  hab468bb_0              conda-forge         Cached
  + krb5                             1.20.1  h127bd45_0              conda-forge         Cached
  + libssh2                          1.10.0  hb80f160_3              conda-forge         Cached
  + libnghttp2                       1.51.0  hd184df1_0              conda-forge         Cached
  + libgettextpo                     0.22.5  h8414b35_3              conda-forge         Cached
  + gettext-tools                    0.22.5  h8414b35_3              conda-forge         Cached
  + libintl-devel                    0.22.5  h8414b35_3              conda-forge         Cached
  + libglib                          2.80.3  h59d46d9_2              conda-forge         Cached
  + libgfortran                       5.0.0  13_2_0_hd922786_3       conda-forge         Cached
  + brotli-bin                        1.1.0  hb547adb_1              conda-forge         Cached
  + fontconfig                       2.14.2  h82840c6_0              conda-forge         Cached
  + libclang                         14.0.6  default_h5dc8d65_1      conda-forge         Cached
  + lcms2                              2.14  h8193b64_0              conda-forge         Cached
  + openjpeg                          2.5.0  h5d4e404_1              conda-forge         Cached
  + grpc-cpp                         1.46.3  hacd037c_3              conda-forge         Cached
  + libxslt                          1.1.37  h1bd8bc4_0              conda-forge         Cached
  + mysql-libs                       8.0.32  hea58576_0              conda-forge         Cached
  + libpq                              15.1  hbce9e56_3              conda-forge         Cached
  + libcurl                          7.87.0  hbe9bab4_0              conda-forge         Cached
  + libgettextpo-devel               0.22.5  h8414b35_3              conda-forge         Cached
  + glib-tools                       2.80.3  h8ba3eef_2              conda-forge         Cached
  + fftw                             3.3.10  nompi_h6637ab6_110      conda-forge         Cached
  + libopenblas                      0.3.27  openmp_h517c56d_1       conda-forge         Cached
  + brotli                            1.1.0  hb547adb_1              conda-forge         Cached
  + hdf5                             1.12.2  nompi_h55deafc_101      conda-forge         Cached
  + gettext                          0.22.5  h8414b35_3              conda-forge         Cached
  + openblas                         0.3.27  openmp_h560b219_1       conda-forge         Cached
  + libblas                           3.9.0  23_osxarm64_openblas    conda-forge         Cached
  + libidn2                           2.3.7  h93a5062_0              conda-forge         Cached
  + libcblas                          3.9.0  23_osxarm64_openblas    conda-forge         Cached
  + liblapack                         3.9.0  23_osxarm64_openblas    conda-forge         Cached
  + gnutls                            3.7.9  hd26332c_0              conda-forge         Cached
  + liblapacke                        3.9.0  23_osxarm64_openblas    conda-forge         Cached
  + blas-devel                        3.9.0  23_osxarm64_openblas    conda-forge         Cached
  + blas                              2.123  openblas                conda-forge         Cached
  + tzdata                            2024a  h8827d51_1              conda-forge         Cached
  + font-ttf-dejavu-sans-mono          2.37  hab24e00_0              conda-forge         Cached
  + font-ttf-inconsolata              3.000  h77eed37_0              conda-forge         Cached
  + font-ttf-source-code-pro          2.038  h77eed37_0              conda-forge         Cached
  + font-ttf-ubuntu                    0.83  h77eed37_2              conda-forge         Cached
  + fonts-conda-forge                     1  0                       conda-forge         Cached
  + fonts-conda-ecosystem                 1  0                       conda-forge         Cached
  + python                           3.9.15  h2d96c93_0_cpython      conda-forge         Cached
  + cairo                            1.16.0  had492bb_1012           conda-forge         Cached
  + ffmpeg                            4.4.2  gpl_hf318d42_112        conda-forge         Cached
  + harfbuzz                          5.3.0  hddbc195_0              conda-forge         Cached
  + wheel                            0.44.0  pyhd8ed1ab_0            conda-forge         Cached
  + setuptools                       72.2.0  pyhd8ed1ab_0            conda-forge         Cached
  + pip                                24.2  pyh8b19718_1            conda-forge         Cached
  + hyperframe                        6.0.1  pyhd8ed1ab_0            conda-forge         Cached
  + hpack                             4.0.0  pyh9f0ad1d_0            conda-forge         Cached
  + locket                            1.0.0  pyhd8ed1ab_0            conda-forge         Cached
  + pysocks                           1.7.1  pyha2e5f31_6            conda-forge         Cached
  + aiohappyeyeballs                  2.4.0  pyhd8ed1ab_0            conda-forge         Cached
  + pycparser                          2.22  pyhd8ed1ab_0            conda-forge         Cached
  + pyjwt                             2.9.0  pyhd8ed1ab_1            conda-forge         Cached
  + blinker                           1.8.2  pyhd8ed1ab_0            conda-forge         Cached
  + pyasn1                            0.6.0  pyhd8ed1ab_0            conda-forge         Cached
  + fsspec                         2024.6.1  pyhff2d567_0            conda-forge         Cached
  + cachetools                        5.5.0  pyhd8ed1ab_0            conda-forge         Cached
  + zipp                             3.20.1  pyhd8ed1ab_0            conda-forge         Cached
  + idna                                3.8  pyhd8ed1ab_0            conda-forge         Cached
  + charset-normalizer                3.3.2  pyhd8ed1ab_0            conda-forge         Cached
  + toolz                            0.12.1  pyhd8ed1ab_0            conda-forge         Cached
  + cloudpickle                       3.0.0  pyhd8ed1ab_0            conda-forge         Cached
  + threadpoolctl                     3.5.0  pyhc1e730c_0            conda-forge         Cached
  + joblib                            1.4.2  pyhd8ed1ab_0            conda-forge         Cached
  + munkres                           1.1.4  pyh9f0ad1d_0            conda-forge         Cached
  + pyparsing                         3.1.4  pyhd8ed1ab_0            conda-forge         Cached
  + cycler                           0.12.1  pyhd8ed1ab_0            conda-forge         Cached
  + certifi                        2024.7.4  pyhd8ed1ab_0            conda-forge         Cached
  + click                             8.1.7  unix_pyh707e725_0       conda-forge         Cached
  + tensorboard-plugin-wit            1.8.1  pyhd8ed1ab_0            conda-forge         Cached
  + mdurl                             0.1.2  pyhd8ed1ab_0            conda-forge         Cached
  + pygments                         2.18.0  pyhd8ed1ab_0            conda-forge         Cached
  + networkx                            3.2  pyhd8ed1ab_0            conda-forge         Cached
  + jsonpickle                          1.2  py_0                    conda-forge         Cached
  + jsmin                             3.0.1  pyhd8ed1ab_0            conda-forge         Cached
  + attrs                            22.1.0  pyh71513ae_1            conda-forge         Cached
  + pytz                             2024.1  pyhd8ed1ab_0            conda-forge         Cached
  + python-tzdata                    2024.1  pyhd8ed1ab_0            conda-forge         Cached
  + cached_property                   1.5.2  pyha770c72_1            conda-forge         Cached
  + typing_extensions                4.12.2  pyha770c72_0            conda-forge         Cached
  + termcolor                         2.4.0  pyhd8ed1ab_0            conda-forge         Cached
  + six                              1.16.0  pyh6c4a22f_0            conda-forge         Cached
  + python-flatbuffers                 1.12  pyhd8ed1ab_1            conda-forge         Cached
  + packaging                          24.1  pyhd8ed1ab_0            conda-forge         Cached
  + gast                              0.4.0  pyh9f0ad1d_0            conda-forge         Cached
  + absl-py                           2.1.0  pyhd8ed1ab_0            conda-forge         Cached
  + keras                             2.9.0  pyhd8ed1ab_0            conda-forge         Cached
  + h2                                4.1.0  pyhd8ed1ab_0            conda-forge         Cached
  + rsa                                 4.9  pyhd8ed1ab_0            conda-forge         Cached
  + pyasn1-modules                    0.4.0  pyhd8ed1ab_0            conda-forge         Cached
  + importlib-metadata                8.4.0  pyha770c72_0            conda-forge         Cached
  + importlib_resources               6.4.4  pyhd8ed1ab_0            conda-forge         Cached
  + partd                             1.4.2  pyhd8ed1ab_0            conda-forge         Cached
  + markdown-it-py                    3.0.0  pyhd8ed1ab_0            conda-forge         Cached
  + cattrs                            1.1.1  pyhd8ed1ab_0            conda-forge         Cached
  + cached-property                   1.5.2  hd8ed1ab_1              conda-forge         Cached
  + typing-extensions                4.12.2  hd8ed1ab_0              conda-forge         Cached
  + pyu2f                             0.1.5  pyhd8ed1ab_0            conda-forge         Cached
  + python-dateutil                   2.9.0  pyhd8ed1ab_0            conda-forge         Cached
  + google-pasta                      0.2.0  pyhd8ed1ab_1            conda-forge         Cached
  + astunparse                        1.6.3  pyhd8ed1ab_0            conda-forge         Cached
  + qtpy                              2.4.1  pyhd8ed1ab_0            conda-forge         Cached
  + importlib_metadata                8.4.0  hd8ed1ab_0              conda-forge         Cached
  + markdown                            3.6  pyhd8ed1ab_0            conda-forge         Cached
  + importlib-resources               6.4.4  pyhd8ed1ab_0            conda-forge         Cached
  + rich                             13.7.1  pyhd8ed1ab_0            conda-forge         Cached
  + async-timeout                     4.0.3  pyhd8ed1ab_0            conda-forge         Cached
  + brotli-python                     1.1.0  py39hb198ff7_1          conda-forge         Cached
  + multidict                         6.0.5  py39h02fc5c5_0          conda-forge         Cached
  + frozenlist                        1.4.1  py39h17cfd9d_0          conda-forge         Cached
  + markupsafe                        2.1.5  py39h06df861_1          conda-forge         Cached
  + unicodedata2                     15.1.0  py39h0f82c59_0          conda-forge         Cached
  + kiwisolver                        1.4.5  py39h157d57c_2          conda-forge         Cached
  + tensorboard-data-server           0.6.1  py39haa0b8cc_4          conda-forge         Cached
  + pyzmq                            26.2.0  py39h6f9cb01_0          conda-forge         Cached
  + pyyaml                            6.0.2  py39hfea33bf_0          conda-forge         Cached
  + python-rapidjson                   1.20  py39hbf7db11_0          conda-forge         Cached
  + psutil                            6.0.0  py39hfea33bf_0          conda-forge         Cached
  + pillow                            9.2.0  py39h139752e_3          conda-forge         Cached
  + wrapt                            1.16.0  py39h06df861_1          conda-forge         Cached
  + numpy                            1.22.4  py39h7df2422_0          conda-forge         Cached
  + cffi                             1.17.0  py39h7f933ea_1          conda-forge         Cached
  + cytoolz                          0.12.3  py39h17cfd9d_0          conda-forge         Cached
  + protobuf                         3.20.3  py39h23fbdae_1          conda-forge         Cached
  + grpcio                           1.46.3  py39he581682_3          conda-forge         Cached
  + glib                             2.80.3  h59d46d9_2              conda-forge         Cached
  + yarl                              1.9.6  py39h06df861_0          conda-forge          108kB
  + fonttools                        4.53.1  py39hfea33bf_0          conda-forge         Cached
  + imagecodecs-lite              2019.12.3  py39h161d348_8          conda-forge         Cached
  + pywavelets                        1.6.0  py39h161d348_0          conda-forge         Cached
  + contourpy                         1.2.1  py39h48c5dd5_0          conda-forge         Cached
  + shapely                           2.0.6  py39h2abb8a4_0          conda-forge         Cached
  + pandas                            2.2.2  py39h998126f_1          conda-forge         Cached
  + h5py                              3.8.0  nompi_py39hc9149d8_100  conda-forge         Cached
  + libopencv                         4.6.0  py39he1c1adf_3          conda-forge         Cached
  + zstandard                        0.22.0  py39h0b77d07_1          conda-forge         Cached
  + cryptography                     39.0.0  py39haa0b8cc_0          conda-forge         Cached
  + gstreamer                        1.22.9  h551c6ff_1              conda-forge         Cached
  + matplotlib-base                   3.8.4  py39h15359f4_2          conda-forge         Cached
  + py-opencv                         4.6.0  py39hfa6204d_3          conda-forge         Cached
  + gst-plugins-base                 1.22.9  h09b4b5e_1              conda-forge         Cached
  + opencv                            4.6.0  py39hdf13c20_3          conda-forge         Cached
  + qt-main                          5.15.8  hfe8d25c_6              conda-forge         Cached
  + pyside2                          5.15.8  py39h0adaba8_2          conda-forge         Cached
  + aiosignal                         1.3.1  pyhd8ed1ab_0            conda-forge         Cached
  + werkzeug                          3.0.4  pyhd8ed1ab_0            conda-forge         Cached
  + dask-core                      2024.8.0  pyhd8ed1ab_0            conda-forge         Cached
  + patsy                             0.5.6  pyhd8ed1ab_0            conda-forge         Cached
  + imageio                          2.35.1  pyh12aca89_0            conda-forge         Cached
  + opt_einsum                        3.3.0  pyhc1e730c_2            conda-forge         Cached
  + tifffile                       2020.6.3  py_0                    conda-forge         Cached
  + urllib3                           2.2.2  pyhd8ed1ab_1            conda-forge         Cached
  + pyopenssl                        23.2.0  pyhd8ed1ab_1            conda-forge         Cached
  + oauthlib                          3.2.2  pyhd8ed1ab_0            conda-forge         Cached
  + requests                         2.32.3  pyhd8ed1ab_0            conda-forge         Cached
  + requests-oauthlib                 2.0.0  pyhd8ed1ab_0            conda-forge         Cached
  + scipy                             1.7.3  py39haa152ba_2          anaconda            Cached
  + aiohttp                          3.10.5  py39hfea33bf_0          conda-forge         Cached
  + statsmodels                      0.14.1  py39h373d45f_0          conda-forge         Cached
  + scikit-learn                        1.0  py39h12ba089_1          conda-forge         Cached
  + scikit-image                     0.19.3  py39hde7b980_2          conda-forge         Cached
  + seaborn-base                     0.13.2  pyhd8ed1ab_2            conda-forge         Cached
  + pykalman                          0.9.7  pyhd8ed1ab_0            conda-forge         Cached
  + keras-preprocessing               1.1.2  pyhd8ed1ab_0            conda-forge         Cached
  + scikit-video                     1.1.11  pyh24bf2e0_0            conda-forge         Cached
  + google-auth                      2.34.0  pyhff2d567_0            conda-forge         Cached
  + imgaug                            0.4.0  pyhd8ed1ab_1            conda-forge         Cached
  + seaborn                          0.13.2  hd8ed1ab_2              conda-forge         Cached
  + google-auth-oauthlib              0.4.6  pyhd8ed1ab_0            conda-forge         Cached
  + tensorboard                       2.9.0  pyhd8ed1ab_0            conda-forge         Cached
  + tensorflow-base                   2.9.1  cpu_py39ha1ad4ae_0      conda-forge         Cached
  + tensorflow-estimator              2.9.1  cpu_py39h7b621ec_0      conda-forge         Cached
  + tensorflow                        2.9.1  cpu_py39h2839aeb_0      conda-forge         Cached
  + tensorflow-hub                   0.12.0  pyhca92ed8_0            conda-forge         Cached
  + sleap                             1.3.4  py39_0                  sleap/label/dev      240MB

  Summary:

  Install: 253 packages

  Total download: 242MB






⚠️ This build is branched off of main. In order to get the build runners working, we needed to make some changes (that were already made in develop). This will cause merge conflicts when merging develop into main. Anything in develop should override all files in main - even the attrs and opencv constraints added here as they will be more officially handled in develop before the next release.

⚠️ These merge conflicts from develop -> main will break the develop runners from running


Perhaps we release from this branch (not main) and delete this branch instead of merging into main? The source code would still be able to be accessed through the tags and the zipped source code in the release.




Types of changes


    

  •  Bugfix
    • 

  •  New feature
    • 

  •  Refactor / Code style update (no logical changes)
    • 

  •  Build / CI changes
    • 

  •  Documentation Update
    • 

  •  Other (explain)
    • 



Does this address any currently open issues?



Outside contributors checklist


    

  •  Review the guidelines for contributing to this repository
    • 

  •  Read and sign the CLA and add yourself to the authors list
    • 

  •  Make sure you are making a pull request against the develop branch (not main). Also you should start your branch off develop
    • 

  •  Add tests that prove your fix is effective or that your feature works
    • 

  •  Add necessary documentation (if appropriate)
    • 



Thank you for contributing to SLEAP!


❤️

    

  
  



      


          

          
  

    
  
    
    

  

  




          

        
      




  





        

       
            



      

  
        





      

  
      



  

  @coderabbitai

    
      coderabbitai
    




  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  


  

    

      

      
            coderabbitai
  bot

      

      

      commented


        Aug 30, 2024


      
  •

  

    
      

        
          edited
          
        
        
      

    
    
      
        
      Loading

      
    
  



    


  





      


        


  
    

      
          


Important


Review skipped


Auto reviews are disabled on base/target branches other than the default branch.


Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.


You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.







Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?



Share





Tips

Chat


There are 3 ways to chat with CodeRabbit:


    

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:

      

    • I pushed a fix in commit <commit_id>.
      • 

    • Generate unit testing code for this file.
      • 

    • Open a follow-up GitHub issue for this discussion.
      • 

    

    • 

  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:

      

    • @coderabbitai generate unit testing code for this file.
      • 

    • @coderabbitai modularize this function.
      • 

    

    • 

  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:

      

    • @coderabbitai generate interesting stats about this repository and render them as a table.
      • 

    • @coderabbitai show all the console.log statements in this repository.
      • 

    • @coderabbitai read src/utils.ts and generate unit testing code.
      • 

    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
      • 

    • @coderabbitai help me debug CodeRabbit configuration file.
      • 

    

    • 



Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.


CodeRabbit Commands (Invoked using PR comments)


    

  • @coderabbitai pause to pause the reviews on a PR.
    • 

  • @coderabbitai resume to resume the paused reviews.
    • 

  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
    • 

  • @coderabbitai full review to do a full review from scratch and review all the files again.
    • 

  • @coderabbitai summary to regenerate the summary of the PR.
    • 

  • @coderabbitai resolve resolve all the CodeRabbit review comments.
    • 

  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
    • 

  • @coderabbitai help to get help.
    • 



Other keywords and placeholders


    

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
    • 

  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
    • 

  • Add @coderabbitai anywhere in the PR title to generate the title automatically.
    • 



CodeRabbit Configuration File (.coderabbit.yaml)


    

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
    • 

  • Please see the configuration documentation for more information.
    • 

  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
    • 



Documentation and Community


    

  • Visit our Documentation for detailed information on how to use CodeRabbit.
    • 

  • Join our Discord Community to get help, request features, and share feedback.
    • 

  • Follow us on X/Twitter for updates and announcements.
    • 





      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
      



  

  @codecov

    
      Codecov
    




  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  


  

    

      

      
            codecov
  bot

      

      

      commented


        Aug 30, 2024


      
  •

  

    
      

        
          edited
          
        
        
      

    
    
      
        
      Loading

      
    
  



    


  





      


        


  
    

      
          
Codecov Report


All modified and coverable lines are covered by tests ✅




Project coverage is 73.12%. Comparing base (5d71030) to head (d4cc425).

Report is 216 commits behind head on main.




Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1927      +/-   ##
==========================================
+ Coverage   65.97%   73.12%   +7.14%     
==========================================
  Files         127      134       +7     
  Lines       21378    23963    +2585     
==========================================
+ Hits        14105    17522    +3417     
+ Misses       7273     6441     -832     




☔ View full report in Codecov by Sentry.

📢 Have feedback on the report? Share it here.

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
        





      

  
          

  
  

  
      @talmo
  talmo


        marked this pull request as ready for review

    
      September 3, 2024 19:45
    
    









      

  
        





      

  
      



  

  @roomrys





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Collaborator


      

  Author


  


  

    

      

      
            roomrys
  

      

      

      commented


        Sep 3, 2024


      
    


  





      


        


  
    

      
          
We are not merging this PR into main and will be deleting this branch (after 1.4.2 release). The source code will remain available as an asset in the 1.3.4 release notes and also through the v1.3.4 tag.

      		
    

  





        


            

            

              
            

        

      


      

            
  

    
  
    
    

  

  



    












      

  
            

    

      
    

    


      

          @roomrys
roomrys




            closed this


      Sep 3, 2024

    

  


    


  

  

    
  


  

      

  This was referenced Sep 3, 2024





  


  

      
  
  Open

  






      






  


  

      
  
  Closed

  






  









      

  
        

  

      

  
  

  
          

  

    

      


  

      
        @roomrys
  




      

        
          Do not build conda packages on release
        

          
                        
      


      

        

    
    
    

  

      


      

        

          

    
    
    

  

  

    
  



        

      


      

        
          d4cc425
        
      

    

  

    

      
Already built and tested

    







  












  
  

    

      

  




    


    

      

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



      

    

  




  
          




      

  

    
    

    
  

    Reviewers
  


    


        

          

  
      @talmo
  
    talmo



          
            
              
            
          
          talmo approved these changes

        


      






    

  


      
  

    Assignees
  



        

    No one assigned







      


  


  

    Labels
  



    

    None yet







      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      No milestone





    
      
          


  

    
      

        
    

    Development
  



          


Successfully merging this pull request may close these issues.



  


    
  




        
      

    

  
      

  

    

      2 participants
    

    

        
          @roomrys 
        
          @talmo